When we try to reconstruct what a person is seeing based purely on their brain activity, we run into a fundamental optimization trap. If we train a model to map neural signals directly to raw pixels via standard regression, the result is a blurry, unrecognizable smudge. As Umut Güçlü et al. observed in Reconstructing perceived faces from brain activations with deep adversarial neural decoding, pixel-wise loss functions force a model to average out its structural uncertainty, destroying fine details.
To fix this, one might turn to modern generative AI, using high-level semantic embeddings (like CLIP) extracted from the brain to guide an image generator. But this introduces a different failure mode: the model might generate a stunningly realistic photo of a dog, but place it on the wrong side of the screen, at the wrong scale, or with completely incorrect colors. As Changde Du et al. pointed out in MindDiffuser: Controlled Image Reconstruction from Human Brain Activity with Semantic and Structural Diffusion, semantic-only embeddings lack the precise spatial and geometric blueprints required for faithful reconstruction.
The solution that has come to define this research direction is semantic-structural decoupling. By explicitly separating the neural signal into two independent streams—a semantic stream ("what" an object is) and a structural stream ("where" and "how" it is arranged)—decoding models can optimize for category accuracy and spatial fidelity separately before merging them into a final output.
The Biological Inspiration
This functional division is not arbitrary; it directly mimics the human brain's own architecture. In Review of visual neural encoding and decoding methods in fMRI, Changde Du et al. explain that the human visual cortex processes information through parallel pathways.
This mapping corresponds to distinct cortical regions. As Shinji Nishimoto et al. demonstrated in High-resolution image reconstruction with latent diffusion models from human brain activity, signals from the early visual cortex preserve low-level spatial structure, while signals from higher-order cortical areas encode categorical and semantic information. A comprehensive taxonomy by Jian Zhang et al. in Brain-Conditional Multimodal Synthesis: A Survey and Taxonomy confirms that targeting these specific brain regions with distinct generative priors drastically improves reconstruction fidelity.
┌─── Early Visual Cortex ───► Structural Stream (Layout, Color) ──┐
│ ▼
Brain Activity ───┤ Generative Model ──► Reconstruction
│ ▲
└─── Higher Visual Areas ───► Semantic Stream (Category, Text) ───┘
Inside the Two Streams
How do these streams operate in practice?
The semantic stream acts as the conceptual anchor. It translates high-level neural patterns into abstract latent spaces that capture categorical meaning. In Reconstructing the Mind's Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors, Paul S. Scotti et al. used a contrastive learning pipeline to map brain activity directly to a semantic latent space for precise content identification. Similarly, Jiaxuan Chen 0007 et al. in Mind Artist: Creating Artistic Snapshots with Human Thought utilized a linguistic stream to predict text descriptions from neural activity, ensuring the core concept of the image remains stable.
Meanwhile, the structural stream operates in parallel to capture the geometric scaffolding. Rather than predicting pixels, it extracts spatial layouts, silhouettes, depth, or color distributions. For instance, Bohan Zeng et al. developed the framework in Controllable Mind Visual Diffusion Model, which extracts a spatial "silhouette" component from fMRI signals to govern object boundaries. To capture even finer details, Weihao Xia et al. designed the DREAM framework in DREAM: Visual Decoding from REversing HumAn Visual SysteM to explicitly isolate parallel pathways for color and depth, filling the gaps that semantic embeddings inherently miss.
Fusing the Streams with Generative Priors
Once the neural signal is split and decoded into these two distinct blueprints, they are reunited within a generative model—most commonly a Latent Diffusion Model (LDM). The semantic latent acts as a global guide (often via cross-attention mechanisms), while the structural latent acts as a spatial constraint (often via architectures like ControlNet or direct latent initialization).
By feeding these distinct signals into a pre-trained generative model, frameworks like NeuralDiffuser by Haoyu Li et al. in NeuralDiffuser: Neuroscience-Inspired Diffusion Guidance for fMRI Visual Reconstruction achieve a "bottom-up" synthesis. This ensures that the generated image is not just a random realistic depiction of the category, but is tightly anchored to the original visual stimulus's exact layout.
Similarly, Roman Beliy et al. demonstrated in Brain-IT: Image Reconstruction from fMRI via Brain-Interaction Transformer that a dual-branch transformer architecture processing these pathways independently can achieve high-fidelity reconstructions even with highly constrained training data.
Beyond Static Images: Motion and Generalization
The power of the dual-stream paradigm becomes even more apparent when scaling from static images to dynamic videos. In video decoding, the structural stream must expand to capture temporal dynamics without causing the generative model to "hallucinate" erratic changes.
To solve this, Qi Zhang et al. introduced NeuroClips in NeuroClips: Towards High-fidelity and Smooth fMRI-to-Video Reconstruction, which decouples high-level semantic keyframes from low-level perceptual motion flows. By isolating how an object moves from what the object is, the system maintains temporal smoothness. In a similar vein, Yizhuo Lu et al. developed Mind-Animator in Animate Your Thoughts: Reconstruction of Dynamic Natural Vision from Human Brain Activity, mapping brain activity to distinct CLIP (semantic) and VQ-VAE (structural/motion) spaces before integration.
Finally, this decomposition makes neural decoding models remarkably transferable across different subjects and recording devices. Because spatial geometry and high-level semantics are universal human experiences, separating them allows models to generalize better. Shuai Huang et al. applied this dual-pathway strategy to noisy EEG signals in NEED: Cross-Subject and Cross-Task Generalization for Video and Image Reconstruction from EEG Signals, while Yulong Liu et al. leveraged it in See Through Their Minds: Learning Transferable Brain Decoding Models from Cross-Subject fMRI to build cross-subject models that require minimal individual tuning.
Key Foundational Work
Reconstructing perceived faces from brain activations with deep adversarial neural decoding (2017) — contributed 1 concept elements to this direction.
Reconstruction of Natural Visual Scenes from Neural Spikes with Deep Neural Networks (2019) — contributed 1 concept elements to this direction.