Introduction
Reconstructing auditory or imagined speech from brain activity is a foundational goal in Brain-Computer Interface (BCI) research. While invasive methods like Electrocorticography (ECoG) offer high-quality signals, non-invasive Electroencephalography (EEG) is preferred for its safety and accessibility. However, EEG signals suffer from a low signal-to-noise ratio and significant inter-subject variability.
Traditional approaches have focused on simple speech envelope reconstruction or classification tasks. Reconstructing a full mel spectrogram—a detailed frequency-based audio representation—remains a significant challenge, particularly for continuous speech where standard Transformer architectures struggle with the computational complexity of long sequences. To address this, the researchers propose SSM2Mel, a framework leveraging State Space Models (SSMs) for efficient and high-fidelity brain-to-speech decoding.
Methodology
The SSM2Mel architecture moves away from pure Transformer designs, utilizing the linear scaling properties of State Space Models to handle long EEG sequences. The model is composed of four integrated components:
- Embedding Strength Modulator (ESM): To account for the high variability between individuals, the ESM uses subject IDs to dynamically adjust embedding strength, personalizing feature extraction for each participant.
- S4-UNet: This module adapts the U-Net structure—typically used in image segmentation—for time-series data. By replacing standard convolutional layers with Structured State Space Sequence (S4) layers, the model captures multi-scale temporal features more effectively.
- External Attention: This mechanism captures correlations between different samples with linear complexity, avoiding the quadratic overhead of standard self-attention.
- Mamba/MHSA Backbone: The core engine is a hybrid backbone that alternates between Mamba modules (Selective State Space Models) for long-sequence modeling and Multi-Head Self-Attention (MHSA) for global context.
Experimental Setup
The model was evaluated using the SparrKULee dataset from the ICASSP 2024 Auditory EEG Challenge. The dataset includes 85 Dutch-speaking subjects listening to ~15-minute stories in fluent Flemish. Preprocessing involved multichannel Wiener filtering for artifact removal and downsampling to 64 Hz. The data was split into 80% training, 10% validation, and 10% testing sets.
Results and Discussion
The primary performance metric was the Pearson correlation between the reconstructed mel spectrogram and the original audio stimulus. SSM2Mel achieved a correlation of 0.069, representing a 38% improvement over the VLAAI baseline and surpassing previous state-of-the-art models like ConvConcatNet.
Key Insights
- Hybrid Superiority: Ablation studies indicated that the hybrid Mamba/MHSA approach outperforms models using only one of these mechanisms. Mamba excels at temporal signal characteristics, while MHSA captures broader contextual information.
- Personalization: Removing the ESM module led to a significant performance drop, confirming that subject-specific modulation is critical for handling neural variability.
- Efficiency: Mamba's linear scaling proved better suited for continuous EEG sequences than the quadratic scaling of traditional Transformers.
Beyond mel spectrograms, the model also excelled in the simpler speech envelope reconstruction task, achieving a correlation of 0.208 (a 29.2% improvement over previous models).
Conclusion
SSM2Mel demonstrates that State Space Models can significantly advance non-invasive speech reconstruction. While the absolute correlation values remain low—indicating that reconstructed audio is not yet fully intelligible—the 38% improvement over baselines marks a major step forward. Future research will focus on bridging the gap between perceived and purely imagined speech and exploring the feasibility of real-time BCI applications.
Summary updated