Decoding Mental Imagery from Human Brain Activity

39 concepts31 papers

Recent breakthroughs in computational neuroscience have made it possible to reconstruct what a person is looking at or listening to directly from their brain activity. By mapping functional magnetic resonance imaging (fMRI) or electroencephalography (EEG) signals onto generative AI models, frameworks can recreate seen images or heard music with remarkable fidelity.

However, a deeper and more challenging frontier has emerged: decoding mental imagery. Instead of translating passive sensory perception, researchers are attempting to externalize purely internal, endogenous cognitive states—such as voluntary imagination, memories, and dreams—where no external physical stimulus is present.

Understanding why this is possible, and why it remains exceptionally difficult, requires looking at how the brain processes internal thoughts versus external reality.

The Shared Representation Hypothesis

The foundational reason mental imagery decoding works at all is that imagination and perception are not entirely distinct phenomena in the human brain; they share a common neural substrate.

When you look at an object, visual information travels from your retina "bottom-up" into the visual cortex, activating a hierarchy of features from low-level edges to high-level semantic concepts. Crucially, when you close your eyes and merely imagine that same object, the brain repurposes this exact same hierarchical architecture in a "top-down" fashion.

Because of this overlap, models trained exclusively on brain data collected during active viewing can generalize to pure imagination. For example, Guohua Shen et al. demonstrated that a deep image reconstruction framework trained on perception data could decode the rudimentary contents of voluntary mental imagery, confirming that imagination utilizes the same hierarchical feature space as physical sight. Similarly, Rufin VanRullen showed that a Variational Autoencoder-Generative Adversarial Network (VAE-GAN) could map fMRI patterns to a latent space to reconstruct faces that a subject was merely imagining. This principle extends beyond vision: André Ofner et al. built a multi-view generative model that maps EEG signals to a shared musical latent space, allowing them to reconstruct the rhythm and timbre of music that a participant was only imagining.

The Asymmetry of Top-Down Processing

If perception and imagination share the same neural pathways, why can't we seamlessly decode thoughts using existing state-of-the-art perception decoders? The challenge lies in a fundamental physiological asymmetry between bottom-up and top-down processing.

Perception (Bottom-Up):  [External Stimulus] ──> [Retina] ──> [Visual Cortex (High SNR, Stable)]
Imagination (Top-Down):  [Associative Regions] ────────────────> [Visual Cortex (Low SNR, Fleeting)]

External visual stimuli provide stable, high-contrast, and highly structured inputs to the retina. This results in strong, clear neural signals with a high signal-to-noise ratio (SNR). In contrast, mental imagery is driven by high-level associative brain regions. As Shinji Nishimoto points out, internal imagery lacks an external anchor, meaning the resulting neural signals are significantly weaker, more fleeting, and plagued by a much lower SNR.

Furthermore, as Yukiyasu Kamitani notes, internally generated thoughts are inherently less detailed and noisier than direct sensory inputs. This top-down generation is also highly vulnerable to internal cognitive shifts; for instance, Tomoyasu Horikawa et al. showed that subjective attention actively reshapes neural representations, biasing brain activity toward an individual's internal focus rather than an objective state.

The Performance Decoupling Paradox

This difference in signal structure introduces a phenomenon known as performance decoupling: the machine learning architectures that are best at decoding what we see are often poorly suited for decoding what we imagine.

In a comprehensive benchmark using the NSD-Imagery dataset, Reese Kneeland revealed that models optimized for high-fidelity visual reconstruction fail to generalize to mental imagery. Because state-of-the-art vision decoders are trained on the crisp, high-SNR signals of direct perception, they overfit to vision-specific brain patterns and low-level structural details. When forced to process the abstract, noisy signals of pure imagination, these complex models break down.

This structural mismatch was further highlighted by Angeliki Papadimitriou et al.. They discovered a surprising paradox: while deep Multi-Layer Perceptrons (MLPs) excel at decoding perceived images, simpler neighborhood-based models like $k$-Nearest Neighbors ($k$NN) are significantly more effective at decoding mental imagery. This suggests that internally generated neural patterns follow different structural principles, operating at a more abstract, categorical level rather than a precise pixel-level layout.

Overcoming the Conceptual Ground Truth Problem

The final hurdle in decoding the mind's eye is evaluation. When a model reconstructs a perceived image, researchers can mathematically compare the output against the physical image the subject looked at. But when a subject imagines an abstract concept, there is no physical image to serve as an objective ground truth.

To bypass this limitation, modern decoding pipelines abandon the goal of literal pixel-by-pixel reconstruction. Instead, they focus on abstract semantic decoding—mapping brain activity into robust, cross-modal latent spaces (like CLIP embeddings) that capture meaning rather than raw structure.

By aligning noisy neural signals with the semantic spaces of powerful generative models, researchers can synthesize the essence of a thought. For example, Sven Guenther et al. utilized a Latent Diffusion Model paired with low-density EEG to generate recognizable images from pure imagination, trading structural exactness for semantic accuracy. In the auditory domain, Timo I. Denk et al. developed Brain2Music, which maps fMRI responses to semantic embeddings to generate music matching the genre, mood, and instrumentation of the user's internal state.

This shift toward semantic synthesis has transformed mental imagery decoding from a passive observation tool into an interactive medium. Frameworks can now use brain signals as direct prompts for image editing (Muzhou Yu et al.), drive "human-in-the-loop" product design prototypes directly from thought (Pan Wang et al.), and even allow users to manipulate reconstructed dream states using natural language instructions (Yasheng SUN et al.). By treating the brain's internal signals not as broken visual data, but as high-level semantic blueprints, the field is successfully learning to read the unseeable.

Key Foundational Work

Quantitative Modeling of the Visual System (2012) — contributed 1 concept elements to this direction.
Neural Decoding with Hierarchical Generative Models (2010) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_mental_imagery_decoding_3c0564d1,
  title = {mental imagery decoding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mental-imagery-decoding/rdr_3c0564d164a74cd09630ff34c62002d7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_mental_imagery_decoding_3c0564d1_v1,
  title = {mental imagery decoding},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mental-imagery-decoding/rdr_3c0564d164a74cd09630ff34c62002d7/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}

See something wrong? Report a mistake