TL;DR
While AI models have become remarkably good at reconstructing images that a person is actively looking at, they often fail when trying to decode purely imagined thoughts. This paper introduces MIRAGE, a robust multi-modal architecture that prioritizes structural simplicity and text-based guidance to successfully reconstruct internal mental imagery from fMRI brain activity, setting a new state-of-the-art standard.
What Problem Does This Paper Address?
Recent breakthroughs in "vision decoding" allow researchers to reconstruct high-quality images from a subject's functional Magnetic Resonance Imaging (fMRI) data while they view an object. However, the ultimate clinical and practical value of this technology lies in externalizing mental imagery—the private, internal world of our thoughts, imagination, and dreams. This capability is vital for developing communication tools for patients with disorders of consciousness or locked-in syndrome.
The core challenge is that brain activity during imagination is fundamentally different from active vision: it has a much lower signal-to-noise ratio (SNR), lower spatial resolution, and fewer signal dimensions. The authors discover that complex, non-linear models optimized for vision decoding overfit to this noise and completely collapse when tasked with cross-decoding mental images.
Methods
To bridge the gap between vision and imagination, the authors designed MIRAGE (Mental Image Reconstruction using Advanced Generative ModEls) around two main principles: reducing model complexity and leveraging representational alignment.
- Linear Backbone: Instead of using complex Multi-Layer Perceptrons (MLPs), MIRAGE utilizes a simple $L_2$-regularized ridge regression backbone to map fMRI voxels to feature spaces. This prevents the model from overfitting to low-SNR imagery signals.
- Low-Dimensional Image Features: Rather than using high-dimensional spatial patch tokens from CLIP embeddings (which fail under noisy conditions), MIRAGE uses only the compact, low-dimensional CLIP ViT-L/14
CLStoken ($1 \times 768$). - Multi-Modal Text Guidance: Because mental imagery heavily relies on abstract, semantic representations in the brain, the model incorporates long, descriptive synthetic captions generated by a large multimodal model (LLaVA v1.5-13B) to stabilize the diffusion process.
- Dual-Stream Generation: Low-level structural layouts are first decoded using a Very Deep Variational Autoencoder (VDVAE). These initial images are filtered to boost sharpness and contrast before being fed alongside the text and image embeddings into a state-of-the-art diffusion model (Stable Cascade).
- Retrieval Pooling: The system generates 16 candidate images and automatically selects the final output based on the highest cosine similarity to a decoded high-dimensional CLIP embedding.
Datasets
- Natural Scenes Dataset (NSD): Used exclusively for training. It consists of 7T fMRI responses from subjects viewing up to 30,000 natural images sourced from MS COCO.
- NSD-Imagery Dataset: Used as the evaluation benchmark. It contains fMRI data from the same subjects as they vividly imagine 18 distinct stimuli across three categories: Simple (geometric shapes), Complex (natural scenes), and Conceptual (abstract single-word concepts like "zebra" or "banana").
Key Insights and Ideas
- SOTA Vision $\neq$ SOTA Imagery: High performance on seen image reconstruction does not guarantee success on mental imagery. Models like MindEye2 achieve top-tier vision results but fail on imagery trials due to architectural fragility.
- Language Anchors the Imagination: Brain activity during mental imagery aligns closely with language-like representations in higher-level visual areas. Incorporating high-quality text embeddings allows the generative model to maintain semantic consistency even when fine-grained visual signals are degraded.
- Ablation Takeaway: Sparser image embeddings provide better utility in low-SNR retrieval contexts. Complex architectures are a disadvantage when attempting to generalize vision performance to mental imagery.
Key Results
MIRAGE establishes a new state-of-the-art for mental image reconstruction across both automated metrics and large-scale human behavioral studies ($n=500$ raters).
- Human Identification Accuracy: In a 2-alternative forced choice (2AFC) task, human raters correctly matched MIRAGE's mental imagery reconstructions to the ground truth 78.30% of the time across all stimuli, compared to just 56.96% for MindEye2 (where chance is 50%).
- Qualitative Superiority: MIRAGE successfully captured structural layouts for simple shapes and generated highly accurate semantic categories for complex and conceptual prompts, outperforming previous methods like Brain Diffuser and iCNN.
Limitations and Future Work
- Dataset Prior Bias: MIRAGE is heavily influenced by the natural image prior of the NSD training set. Consequently, it struggles with out-of-distribution shapes or sparse categories, often filling in fine details with random semantic content from its training distribution.
- Benchmark Scale: The NSD-Imagery dataset is small (only 18 stimuli), preventing large-scale model fine-tuning directly on mental imagery data.
- Compute Requirements: Training requires substantial hardware resources, including 512GB of RAM for the ridge regression modules and high-end GPUs (NVIDIA A100) for inference.
Summary updated