Beyond Alignment: Supervising the Generator for EEG-to-Image Reconstruction

Sangmin Jo, Wootaek Jeong, Heung-Il SukarXiv 2026 · arXiv

TL;DR

Researchers have discovered that simply "aligning" EEG signals with image features isn't enough to reconstruct high-quality images from brain activity. To bridge this gap, they introduced Generation-Derived Supervision (GDS), a method that uses an image-conditioned "teacher" model to guide the internal denoising process of an EEG-conditioned "student" generator, achieving new state-of-the-art results in visual brain decoding.

What Problem Does This Paper Address?

Visual brain decoding—translating neural activity into images—is a holy grail for brain-computer interfaces (BCIs). While Electroencephalography (EEG) is portable and non-invasive, its low signal-to-noise ratio makes reconstruction incredibly difficult.

Most current research focuses on cross-modal alignment: training an encoder to make EEG signals "look like" pretrained image embeddings (e.g., CLIP or DINO features). The assumption was that better alignment would automatically lead to better images. However, the authors demonstrate an "alignment paradox": even when retrieval accuracy (finding the matching image in a database) improves by up to 77%, the actual quality of the generated images often stagnates.

The authors argue that the bottleneck isn't just the input signal; it’s the generative process itself. Standard image generators struggle to follow "static" features from a vision encoder. There is a fundamental mismatch between the rigid output of a vision encoder and the dynamic, evolving internal representations a diffusion model needs during its denoising trajectory.

Methods

The authors propose a two-stage framework built on Scalable Interpolant Transformers (SiT) and Flow Matching:

  1. Stage 1: Image-Conditioned Teacher Training. They first train a "teacher" SiT model to reconstruct images using high-quality, static features from a frozen DINOv2 encoder. This teacher learns how to translate "rigid" vision features into the "fluid" internal representations required for image generation.
  2. Stage 2: EEG-Conditioned Student Training. They then train a "student" SiT model conditioned on EEG signals. Crucially, they don't just ask the student to match the final image; they force the student’s intermediate layers to align with the teacher’s intermediate layers using a cosine similarity loss.
  3. Generation-Derived Supervision (GDS): By using the teacher's internal "thoughts" as a moving target, the student learns a denoising trajectory that is "reachable" and native to the generative architecture, rather than trying to mimic a static external encoder.

At inference time, the teacher is discarded, leaving a lightweight student model that can reconstruct images directly from EEG.

Datasets

The model was evaluated on two major public benchmarks for neural decoding:

  • THINGS-EEG: A large-scale dataset featuring 10 participants viewing 1,654 object concepts. It uses 17 electrodes focused on the occipital and parietal (visual) regions.
  • THINGS-MEG: A similar dataset using Magnetoencephalography (MEG) from 4 participants with 271 sensors, offering higher spatial resolution than EEG.

Key Insights and Ideas

  • The Alignment Bottleneck: The paper proves that "better alignment $\neq$ better reconstruction." Improving the conditioning signal alone has diminishing returns if the generator isn't supervised on how to use that signal.
  • Dynamic vs. Static Targets: Static vision encoders (like DINO) provide discriminative features, but they don't "speak the language" of the denoising process. The teacher model acts as a translator, turning static visual information into a generative "trajectory."
  • Early-to-Late Alignment: Through ablation, the authors found that aligning an early student block with a later teacher block is most effective. This allows discriminative information to propagate through the student's network and influence the entire generation process.
  • Mechanistic Superiority: Using Linear CKA (Centered Kernel Alignment) analysis, the authors showed that their student model actually develops internal representations that are more "visually structured" than models trained with direct DINO alignment.

Key Results

The proposed method achieved state-of-the-art (SOTA) performance across nearly all metrics:

  • High-Level Fidelity: On THINGS-EEG, the model achieved a CLIP score of 0.799 and the lowest SwAV distance (0.500), indicating superior semantic accuracy.
  • Low-Level Detail: It outperformed baselines in PixCorr (0.263) and SSIM (0.370), showing it better preserves the shapes and colors of the original stimuli.
  • Robustness: The GDS framework improved performance regardless of which EEG encoder was used (e.g., Brain-HiVE, SAMGA, or UBP), proving it is a versatile "plug-and-play" supervision strategy.
  • Qualitative Gains: Visual results show the model is significantly better at capturing the specific category and spatial layout of objects (e.g., correctly reconstructing the silhouette of a bird or the structure of a car) compared to previous methods that often produced "blobs" of the correct color.

Limitations and Future Work

  • The Retrieval-Reconstruction Gap: While this paper narrows the gap, the authors acknowledge that the specific visual cues that help a model identify an image aren't always the same ones needed to reconstruct it.
  • Subject-Specific Tuning: Currently, the generator must be fine-tuned for each individual participant. Future work could explore "universal" brain decoders that generalize across subjects without retraining.
  • Computational Cost: Full fine-tuning of a Diffusion Transformer for every subject is computationally expensive compared to methods that only train a small linear probe or adapter.

Summary updated

Abstract

Visual brain decoding translates neural activity into human-interpretable images. While recent EEG-to-image reconstruction methods rely on aligning EEG signals with pretrained visual embeddings to condition an image generator, we empirically demonstrate that stronger cross-modal alignment do not translate into comparable gains in reconstruction quality. This indicates that improving the input condition alone is insufficient; the generative process itself requires explicit supervision. Although Representation Alignment (REPA) offers a solution by guiding diffusion transformers with pretrained vision encoder features, we find that EEG-conditioned generators struggle to follow these fixed targets. To address this mismatch, we introduce generation-derived supervision (GDS): an image-conditioned teacher that translates static image-encoder features into the internal representation space of a diffusion transformer. Our EEG-conditioned student generator closely tracks these dynamic targets during denoising, discarding the teacher at inference. Evaluated on THINGS-EEG and THINGS-MEG, our approach consistently outperforms conditioning-centric baselines, achieving state-of-the-art reconstruction fidelity.
BibTeX
@misc{jo2026beyond,
  author = {Sangmin Jo and Wootaek Jeong and Heung-Il Suk},
  title = {{Beyond Alignment: Supervising the Generator for EEG-to-Image Reconstruction}},
  year = {2026},
  eprint = {2611.19232},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19232},
}
Figure