Introduction
Reconstructing visual experiences from brain activity—specifically functional Magnetic Resonance Imaging (fMRI)—represents a significant frontier in brain-computer interfaces and cognitive neuroscience. While recent advancements leveraging powerful generative models like Stable Diffusion have produced high-fidelity reconstructions, they often treat the brain as a monolithic "black box." This approach ignores the hierarchical and specialized nature of the human visual system and creates a significant barrier to cross-subject generalization, typically requiring massive amounts of data to train bespoke models for every individual.
The Challenge: Interpretability and Generalization
Existing neural decoding models face two primary bottlenecks. First, they lack interpretability; it is difficult to discern how a model translates 3D voxels of brain activity into semantic image features. Second, because brain topography varies across individuals, models struggle to generalize. Most state-of-the-art systems require training from scratch for each new subject, which is computationally expensive and data-intensive.
Methodology: A Biologically Inspired Architecture
To address these limitations, the authors propose MoRE-Brain (Mixture-of-Experts Routed Brain decoder). This framework explicitly mirrors the biological structure of the visual cortex through a hierarchical Mixture-of-Experts (MoE) architecture.
The framework operates in two distinct stages:
- Hierarchical MoE fMRI Encoder: Instead of a single network, MoRE-Brain utilizes a hierarchy of trainable "routers" that assign specific brain regions to specialized "experts" (subnetworks). These experts translate brain signals into the CLIP embedding space across four levels of increasing granularity.
- Dynamically Conditioned Image Generation: The resulting embeddings condition a fine-tuned Stable Diffusion XL (SDXL) model. This is managed by a dual-routing mechanism: a Time Router that selects the appropriate hierarchy level for different diffusion steps, and a Space Router that modulates where expert features are applied spatially on the generated image.
For cross-subject generalization, MoRE-Brain assumes that core visual computations (the "experts") are universal. To adapt to a new subject, the experts remain frozen while only the routers are fine-tuned to map the new subject's unique brain topography to the shared expert pool.
Results and Analysis
MoRE-Brain achieves visual reconstruction quality that is highly competitive with current state-of-the-art models like MindEye2 and MindBridge, performing well across both low-level (SSIM) and high-level (InceptionV3, DreamSim) metrics.
Key Insights
- Emergent Specialization: The MoE architecture leads to artificial experts that naturally mimic biological brain regions. Deeper experts specialized in specific semantic categories, such as "outdoor" scenes, aligning with the brain's dorsal attention network.
- Biologically Plausible Processing: The Time Router autonomously learned a coarse-to-fine schedule, prioritizing lower-level global features during early diffusion steps and shifting to higher-level details in the final stages.
- Efficient Adaptation: By freezing experts and only training routers, the model reduces trainable parameters by over 55%. It can successfully generalize to new subjects using as little as 2.5% to 10% of standard training data (1 to 4 fMRI sessions).
Discussion and Limitations
While MoRE-Brain offers a more interpretable and efficient path forward, it is not without limitations. The model relies on fMRI BOLD signals, which are inherently limited by temporal blurring (measuring blood flow rather than millisecond-level neural spikes). Additionally, the current input is restricted to the visual cortex, meaning the model decodes what is seen rather than higher-level cognitive thoughts or imagination. Future work may involve moving away from pre-trained diffusion priors toward autoregressive models trained directly on brain-to-pixel mappings to further reduce generative bias.
Summary updated