Imagine looking at a photograph of a red sports car parked on a cobblestone street. As your eyes take in the scene, your brain processes two distinct types of information simultaneously: the low-level geometry (the sharp edges of the chassis, the texture of the stones, the exact coordinates of the wheels) and the high-level semantics (the concept of a "car," "speed," or "luxury").
If a scientist tries to reconstruct this image directly from your brain activity, they run into a fundamental physical limitation. Non-invasive neural recording techniques, such as fMRI or EEG, are incredibly noisy and lack the spatial or temporal resolution to capture every pixel-level detail. Early decoding approaches, such as those by Shen et al., attempted to map brain activity directly to low-level visual features, which often resulted in blurry, unrecognizable averages.
To overcome this, modern neural decoders leverage the power of Generative AI. By pairing brain signals with deep generative models like Generative Adversarial Networks (GANs) or Latent Diffusion Models (LDMs), researchers can synthesize crisp, photorealistic outputs. The brain signal no longer needs to supply every pixel; instead, it acts as a high-level "prompt" that guides a creative engine.
However, this marriage of neuroscience and generative modeling introduces a profound paradox: the semantic-structural fidelity gap. While the resulting images look beautiful, how much of the output is a genuine reflection of the subject's thoughts, and how much is merely a clever guess by the AI?
Gist vs. Geometry: Why the Brain Favors Semantics
The root of the fidelity gap lies in how abstract information is organized in the human brain. High-level conceptual meaning—often called the semantic "gist"—leaves a broad, robust footprint across the cortex. Consequently, non-invasive decoders are highly effective at extracting what a person is experiencing, even when they fail to capture the precise physical structure.
This phenomenon spans multiple sensory domains:
- Language: When decoding inner speech or perceived text from EEG signals, Wang et al. observed that systems successfully recover high-level semantic concepts and entity types even when they fail to reconstruct exact surface-level words. The neural signal preserves the meaning, but the exact lexical tokens are lost in the noise.
- Video: In reconstructing dynamic visual experiences, Chen et al. achieved an impressive 85% semantic category accuracy. Yet, the structural similarity to the original video remained significantly lower, demonstrating that high-level meaning is more robustly preserved in neural signals than fine-grained spatial layouts.
- Images: As Li et al. noted, neural decoders excel at identifying the conceptual category of an object, but they struggle to replicate precise low-level visual geometry.
Because generative models are explicitly trained to output sharp, coherent data, they eagerly fill this structural void using their own internal knowledge.
The Illusion of Fidelity: Generative Hallucination
When a generative model receives an incomplete or noisy semantic prompt from a brain decoder, it relies on its generative priors—the statistical patterns it learned during training—to fill in the missing details. This leads to what researchers call generative hallucination.
Sobotka et al. describe this as a "Goldilocks" problem. Simple decoding models are faithful but produce blurry, low-detail reconstructions. Powerful generative models produce beautiful images, but they introduce plausible yet entirely incorrect details that were never present in the original stimulus.
[Noisy Brain Signal] ──> Extracts Semantic Gist ──> "Giraffe"
│
▼
[Generative Prior] ───> Hallucinates Context ────> Adds Savannah Grass
(Not in original image)
This bias can completely obscure the true performance of a brain-computer interface. For instance, Xia pointed out that if a subject views a picture of a giraffe against a blank white wall, a diffusion-based decoder might output a photorealistic giraffe standing in savannah grass. The image looks like a triumph of mind-reading, but the savannah grass is a hallucination born from the model's prior training ("giraffes belong on savannas"), not the subject's neural data.
As WEI et al. argue, this excessive reliance on generative priors makes it incredibly difficult to verify whether an output is a genuine decoding of neural signals or merely a highly sophisticated guess. To combat this, Qian et al. proposed a rigorous litmus test for decoding fidelity: if a model is truly grounded in biological data, it should produce non-semantic noise when fed random input, rather than spinning up a beautiful, hallucinated image out of thin air.
Reconciling Meaning and Structure
To close the semantic-structural gap, recent research has shifted away from monolithic generation toward architectures that explicitly decouple and then realign these competing pathways. The goal is to force the generative model to respect the strict anatomical boundaries decoded from the brain.
One prominent strategy is multi-pathway decomposition. For example, the DREAM framework introduced by Xia explicitly deciphers parallel pathways for semantics, color, and depth from fMRI data. By treating geometry and meaning as separate streams, the model prevents a high-level category (like "car") from distorting the actual spatial coordinates of the object. Similarly, in video reconstruction, Lu et al. independently decode semantic, structural, and motion features to ensure that the reconstructed movement genuinely reflects the brain's perception rather than the AI's internal predictions.
Another approach is coarse-to-fine hierarchical alignment. In the auditory domain, Du et al. and Liu 0002 et al. designed a framework that mirrors the human auditory processing pathway. The system first decodes the high-level semantic meaning of a sound to establish a robust conceptual sketch. This sketch then acts as a constraint, guiding a latent diffusion model as it reconstructs the fine-grained, low-level acoustic and spectral details.
By treating generative AI not as an unsupervised artist, but as a constrained renderer guided by independent structural and semantic neural blueprints, modern decoding frameworks are beginning to achieve outputs that are both visually stunning and biologically faithful.
Key Foundational Work
Generative Adversarial Networks Conditioned by Brain Signals (2017) — contributed 2 concept elements to this direction.