MoRE-Brain: Routed Mixture of Experts for Interpretable and Generalizable Cross-Subject fMRI Visual Decoding

Yuxiang Wei, Yanteng Zhang, Xi Xiao, Tianyang Wang, Xiao Wang, Vince D. Calhoun

Introduction

Reconstructing visual experiences from brain activity—specifically functional Magnetic Resonance Imaging (fMRI)—represents a significant frontier in brain-computer interfaces and cognitive neuroscience. While recent advancements leveraging powerful generative models like Stable Diffusion have produced high-fidelity reconstructions, they often treat the brain as a monolithic "black box." This approach ignores the hierarchical and specialized nature of the human visual system and creates a significant barrier to cross-subject generalization, typically requiring massive amounts of data to train bespoke models for every individual.

The Challenge: Interpretability and Generalization

Existing neural decoding models face two primary bottlenecks. First, they lack interpretability; it is difficult to discern how a model translates 3D voxels of brain activity into semantic image features. Second, because brain topography varies across individuals, models struggle to generalize. Most state-of-the-art systems require training from scratch for each new subject, which is computationally expensive and data-intensive.

Methodology: A Biologically Inspired Architecture

To address these limitations, the authors propose MoRE-Brain (Mixture-of-Experts Routed Brain decoder). This framework explicitly mirrors the biological structure of the visual cortex through a hierarchical Mixture-of-Experts (MoE) architecture.

Overview of the MoRE-Brain two-stage training framework.

The framework operates in two distinct stages:

  1. Hierarchical MoE fMRI Encoder: Instead of a single network, MoRE-Brain utilizes a hierarchy of trainable "routers" that assign specific brain regions to specialized "experts" (subnetworks). These experts translate brain signals into the CLIP embedding space across four levels of increasing granularity.
  2. Dynamically Conditioned Image Generation: The resulting embeddings condition a fine-tuned Stable Diffusion XL (SDXL) model. This is managed by a dual-routing mechanism: a Time Router that selects the appropriate hierarchy level for different diffusion steps, and a Space Router that modulates where expert features are applied spatially on the generated image.

For cross-subject generalization, MoRE-Brain assumes that core visual computations (the "experts") are universal. To adapt to a new subject, the experts remain frozen while only the routers are fine-tuned to map the new subject's unique brain topography to the shared expert pool.

Results and Analysis

MoRE-Brain achieves visual reconstruction quality that is highly competitive with current state-of-the-art models like MindEye2 and MindBridge, performing well across both low-level (SSIM) and high-level (InceptionV3, DreamSim) metrics.

Qualitative comparison of visual reconstructions across different models.

Key Insights

  • Emergent Specialization: The MoE architecture leads to artificial experts that naturally mimic biological brain regions. Deeper experts specialized in specific semantic categories, such as "outdoor" scenes, aligning with the brain's dorsal attention network.
  • Biologically Plausible Processing: The Time Router autonomously learned a coarse-to-fine schedule, prioritizing lower-level global features during early diffusion steps and shifting to higher-level details in the final stages.
  • Efficient Adaptation: By freezing experts and only training routers, the model reduces trainable parameters by over 55%. It can successfully generalize to new subjects using as little as 2.5% to 10% of standard training data (1 to 4 fMRI sessions).

Discussion and Limitations

While MoRE-Brain offers a more interpretable and efficient path forward, it is not without limitations. The model relies on fMRI BOLD signals, which are inherently limited by temporal blurring (measuring blood flow rather than millisecond-level neural spikes). Additionally, the current input is restricted to the visual cortex, meaning the model decodes what is seen rather than higher-level cognitive thoughts or imagination. Future work may involve moving away from pre-trained diffusion priors toward autoregressive models trained directly on brain-to-pixel mappings to further reduce generative bias.

Summary updated

Abstract

Decoding visual experiences from fMRI offers a powerful avenue to understand human perception and develop advanced brain-computer interfaces. However, current progress often prioritizes maximizing reconstruction fidelity while overlooking interpretability, an essential aspect for deriving neuroscientific insight. To address this gap, we propose MoRE-Brain, a neuro-inspired framework designed for high-fidelity, adaptable, and interpretable visual reconstruction. MoRE-Brain uniquely employs a hierarchical Mixture-of-Experts architecture where distinct experts process fMRI signals from functionally related voxel groups, mimicking specialized brain networks. The experts are first trained to encode fMRI into the frozen CLIP space. A finetuned diffusion model then synthesizes images, guided by expert outputs through a novel dual-stage routing mechanism that dynamically weighs expert contributions across the diffusion process. MoRE-Brain offers three main advancements: First, it introduces a novel Mixture-of-Experts architecture grounded in brain network principles for neuro-decoding. Second, it achieves efficient cross-subject generalization by sharing core expert networks while adapting only subject-specific routers. Third, it provides enhanced mechanistic insight, as the explicit routing reveals precisely how different modeled brain regions shape the semantic and spatial attributes of the reconstructed image. Extensive experiments validate MoRE-Brain's high reconstruction fidelity, with bottleneck analyses further demonstrating its effective utilization of fMRI signals, distinguishing genuine neural decoding from over-reliance on generative priors. Consequently, MoRE-Brain marks a substantial advance towards more generalizable and interpretable fMRI-based visual decoding. Code will be publicly available soon: https://github.com/yuxiangwei0808/MoRE-Brain.
BibTeX
@inproceedings{wei2025more,
  author = {Yuxiang Wei and Yanteng Zhang and Xi Xiao and Tianyang Wang and Xiao Wang and Vince D. Calhoun},
  title = {{MoRE-Brain: Routed Mixture of Experts for Interpretable and Generalizable Cross-Subject fMRI Visual Decoding}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=fYSPRGmS6l},
}
Figure