Abstract
Decoding linguistic information from electroencephalography (EEG) remains challenging due to the brain's distributed and nonlinear organization. We present BrainStack, a functionally guided neuro-mixture-of-experts (Neuro-MoE) framework that models the brain's modular functional architecture through anatomically partitioned expert networks. Each functional region is represented by a specialized expert that learns localized neural dynamics, while a transformer-based global expert captures cross-regional dependencies. A learnable routing gate adaptively aggregates these heterogeneous experts, enabling context-dependent expert coordination and selective fusion. To promote coherent representation across the hierarchy, we introduce cross-regional distillation, where the global expert provides top-down regularization to the regional experts. We further release SilentSpeech-EEG (SS-EEG), a large-scale benchmark comprising over 120 hours of EEG recordings from 12 subjects performing 24 silent words, the largest dataset of its kind. Experiments demonstrate that BrainStack consistently outperforms state-of-the-art models, achieving superior accuracy and generalization across subjects. Our results establish BrainStack as a functionally modular, neuro-inspired MoE paradigm that unifies neuroscientific priors with adaptive expert routing, paving the way for scalable and interpretable brain-language decoding.
Introduction
Extracting meaningful language data from electroencephalography (EEG) signals is a cornerstone of non-invasive Brain-Computer Interfaces (BCIs). However, EEG signals are notoriously noisy, non-stationary, and exhibit high inter-subject variability. Traditional deep learning approaches, such as standard CNNs or Transformers, often treat the entire electrode array as a homogeneous input. This monolithic treatment ignores the brain's fundamental modularity—the fact that different cortical regions (e.g., prefrontal, temporal, occipital) are specialized for distinct cognitive subfunctions. By failing to exploit these specialized dynamics, existing models often overfit to whole-brain noise and struggle to generalize across individuals.
The BrainStack Framework
To bridge the gap between neural architectures and neuroscientific reality, we introduce BrainStack, a functionally guided "Neuro-Mixture-of-Experts" (Neuro-MoE) framework. BrainStack partitions EEG inputs into seven canonical anatomical regions, assigning each to a specialized expert network.
The architecture consists of three primary components:
- Local Experts (CNet): Lightweight CNNs designed to capture localized, short-duration spatiotemporal dynamics within specific anatomical partitions.
- Global Expert (CTNet): A hybrid CNN-Transformer that processes the entire cortical coverage to capture long-range dependencies and provide cross-regional context.
- Adaptive Routing Gate: A learnable mechanism that dynamically assigns weights to the representations from both global and local experts. This allows the model to selectively focus on the most informative regions for a given trial.
To ensure hierarchical coherence, we employ cross-regional distillation. During training, a Kullback-Leibler (KL) divergence loss aligns the output distributions of local experts with the global expert, providing "top-down" semantic guidance to the regional networks.
Experimental Setup and Results
We evaluated BrainStack on SilentSpeech-EEG (SS-EEG), a newly introduced benchmark and the largest of its kind. SS-EEG contains over 120 hours of recordings from subjects performing covert articulation of 24 English words.
BrainStack achieved a state-of-the-art average classification accuracy of 41.87% across 10 subjects, significantly outperforming the strongest baseline, TCNet (29.50%), by a margin of +12.37%. Notably, BrainStack also outperformed LaBraM, a large-scale pretrained EEG foundation model, despite having only one-fifth of the parameters (1.06M vs 5.80M). This suggests that neuro-inspired architectural design can be more effective than brute-force scaling for low-SNR neural decoding tasks.
Key Insights and Interpretability
A significant advantage of the Neuro-MoE approach is its inherent interpretability. By analyzing the adaptive routing gate weights, we can observe how the model prioritizes different brain regions based on the task context.
Our analysis revealed that the Occipital (visual) expert showed a strong positive correlation with subject-level accuracy. This suggests that visual-cortical activity—likely related to reading prompts or mental imagery—plays a crucial role in silent speech decoding. Furthermore, the routing gate acts as a personalized filter, leaning on stable regions (like the Temporal and Occipital lobes) for high-performing subjects while attenuating noise in others, which is critical for managing human neural heterogeneity.
Conclusion
BrainStack demonstrates that incorporating anatomical priors into deep learning architectures significantly enhances the decoding of complex neural signals. By combining localized expertise with global context through an adaptive routing mechanism, BrainStack achieves superior accuracy and efficiency compared to monolithic models. This modular paradigm offers a scalable and interpretable path forward for the next generation of language-based BCIs.
Summary updated