Hierarchical Dual Stream Visual Decoding from Neural Signals

This direction is merged into another direction: cross-subject generalization, dual-stream neural decoding. Its last version stays readable and citable here.

Version 1 of 148 papers66 conceptsfrozen 2026-10-04

Reconstructing what a person sees from their brain activity is a classic "inverse problem." We observe the noisy, low-resolution electrical or blood-oxygen signals (EEG or fMRI) and try to work backward to the high-fidelity image that caused them. Early attempts at this were "flat"—they tried to map neural voxels directly to pixels or simple image features.

However, recent breakthroughs have shifted toward Hierarchical Dual-Stream Decoding. Instead of treating the brain as a black box, researchers like Guohua Shen and Tomoyasu Horikawa have shown that the most effective way to "read" the mind is to mimic the functional organization of the human visual cortex itself.

The Ventral and Dorsal Split

The human brain does not process an image as a single block of pixels. It immediately bifurcates information into two primary pathways: the ventral stream (the "what" pathway) for identifying objects and categories, and the dorsal stream (the "where" pathway) for spatial layout, motion, and depth.

Modern architectures now explicitly mirror this split. For instance, the MindEye model developed by Paul S. Scotti and colleagues uses a dual-pipeline architecture. One path maps neural signals to a semantic latent space (like CLIP) to identify what is in the image, while a parallel path maps to a variational autoencoder (VAE) space to preserve where things are.

By decoupling these tasks, models avoid the "blurry blob" problem. If a model tries to decode semantics and structure simultaneously from the same signal, the noise in the spatial data often degrades the categorical accuracy. As Changde Du notes, separating these streams allows for independent optimization: the model can be "sure" it's looking at a cat (ventral) even if it's still refining exactly where the cat's ears are (dorsal).

Hierarchical Decomposition of Visual Primitives

Beyond the "what vs. where" split, the brain processes information in a tiered hierarchy. The early visual cortex (V1, V2) detects simple edges and textures, while mid-level areas like V4 handle curves and parts, and higher areas handle full objects and scenes.

The ViEEG framework by Minxu Liu and Donghai Guan leverages this by decomposing stimulus images into hierarchical layers—contours, foreground objects, and full scenes—before training. By aligning the decoding model's internal layers with these biological stages, they achieved a 50% performance boost over flat models.

This hierarchical approach works because it provides a "scaffold" for the reconstruction. As seen in the DREAM framework by Weihao Xia, the model first establishes a structural silhouette (the "scaffold") and then "paints" the semantic and color details onto it. This prevents the generative model from hallucinating objects in the wrong places.

Generative Models as Neural Orchestrators

The final "trick" that makes modern decoding work is the use of Latent Diffusion Models (LDMs) as the rendering engine. Neural signals are too noisy to provide every pixel of an image. Instead, researchers use the brain signals as "prompts" or "constraints" for a pre-trained generative model like Stable Diffusion.

In the NeuroPictor system by Jingyang Huo and Yikai Wang, the decoded semantic features act as a global guide, while decoded structural features act as a local "ControlNet." This allows the AI to fill in the gaps with realistic textures that the fMRI or EEG might have missed, while still staying faithful to the user's actual perception.

Causal Evidence for Hierarchical Alignment

We know this hierarchical mimicry is actually working—and not just a lucky engineering coincidence—because of "causal masking" experiments. Pinyuan Feng and colleagues demonstrated that if you selectively mask the high-level functional brain regions in their NeuroAdapter model, the semantic objects in the reconstructed image disappear, but the rough layout remains. Conversely, masking low-level regions leaves the "gist" of the object intact but destroys the fine-grained structural details.

This confirms that the models are successfully "hooking into" the specific biological pathways the brain uses to build our visual reality. By reversing the human visual system, these models don't just recognize images; they begin to reconstruct the "mind's eye."

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_hierarchical_dual_stream_visual_decoding_86ed19b4,
  title = {Hierarchical Dual Stream Visual Decoding from Neural Signals},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hierarchical-dual-stream-visual-decoding-from-neural-signals/rdr_86ed19b47d314dab935f3fb68731825e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_hierarchical_dual_stream_visual_decoding_86ed19b4_v1,
  title = {Hierarchical Dual Stream Visual Decoding from Neural Signals},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/hierarchical-dual-stream-visual-decoding-from-neural-signals/rdr_86ed19b47d314dab935f3fb68731825e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}