ConvConcatNet: a deep convolutional neural network to reconstruct mel spectrogram from the EEG

Xiran Xu, Bo Wang, Yujie Yan, Haolin Zhu, Zechen Zhang, Xihong Wu, Jing Chen

TL;DR

Researchers from Peking University have developed ConvConcatNet, a deep learning architecture that reconstructs speech mel spectrograms directly from EEG signals. By using iterative feature concatenation and leveraging speech envelopes as auxiliary training data, the model achieved the top rank in the ICASSP 2024 Auditory EEG Challenge for reconstructing "unseen" stimuli from "unseen" subjects.

What Problem Does This Paper Address?

The "holy grail" of auditory neuroscience is non-invasively reconstructing what a person is hearing based solely on their brain activity (EEG). While we’ve made progress in tracking the simple "envelope" (the overall volume/energy) of speech, reconstructing a full mel spectrogram—which contains rich frequency information—is significantly harder.

The challenge is twofold:

  1. Complexity: The relationship between brain signals and speech is highly non-linear and dynamic. Simple linear models just don't cut it.
  2. Generalization: Most models struggle when faced with "unseen" data—meaning the model must decode a brand-new piece of audio from a person whose brain patterns it has never seen before. This is the ultimate test for any practical brain-computer interface (BCI).

Methods

The authors introduce ConvConcatNet, a model built on three core strategies:

  • Iterative Concatenation: Instead of a standard feed-forward pipeline, the model consists of six blocks that iteratively fuse features. At each step, the original EEG, the output of a "context layer," and the output of a "spatial attention layer" are concatenated. This ensures that spatial-temporal features from different levels of abstraction are preserved throughout the network.
  • The CNN Stack: Each block contains a specialized CNN stack using Sconv (pointwise convolution for spatial features) and Tconv (temporal convolution for time-series features). This allows the model to efficiently process the multi-channel, time-varying nature of EEG.
  • Envelope-Assisted Training: Since previous research shows that EEG tracks the speech envelope quite well, the authors used the envelope as an auxiliary feature during training. By concatenating the envelope with the mel spectrogram during the learning phase, they "guided" the model to find the more complex spectrogram features.
  • Ensemble Learning: To squeeze out maximum performance, they trained an ensemble of 520 models (130 models across 4 cross-validation folds) and averaged their predictions.

Datasets

The study utilized the dataset from the Auditory EEG Challenge (ICASSP 2024):

  • Training Set: EEG responses from 85 young, normal-hearing subjects listening to audiobooks.
  • Testing Set: EEG responses from 20 different subjects (unseen) listening to different stimuli (unseen).
  • Preprocessing: The EEG data was preprocessed and sampled at 64 Hz.

Key Insights and Ideas

  • Feature Persistence: The "aha moment" here is the extensive use of concatenation. By repeatedly feeding the original EEG and intermediate features back into the next block, the network avoids the "vanishing information" problem common in deep architectures, allowing it to maintain a strong link to the raw neural signal.
  • The Envelope as a Bridge: The authors recognized that the mel spectrogram and the speech envelope are highly correlated. Using the envelope as a "hint" during training helps the model bridge the gap between noisy EEG and high-dimensional audio features.
  • Strict Validation for Real-World Use: To prevent the model from simply "memorizing" a specific person's brain quirks or a specific audiobook's rhythm, they used a rigorous data-splitting strategy where the validation set contained entirely different subjects and stimuli than the training set.

Key Results

  • Top Performance: ConvConcatNet achieved a Pearson correlation of 0.0420, ranking No. 1 in Task 2 of the Auditory EEG Challenge.
  • Ablation Success: The full ConvConcatNet outperformed both the baseline model (VLAAI) and a version of itself trained without the speech envelope.
  • Ensemble Boost: The ensembled version significantly improved the correlation coefficient across all test folds, proving that model averaging is a robust strategy for noisy neural data.

Limitations and Future Work

While the model is state-of-the-art, the authors are refreshingly honest about the remaining hurdles:

  • Low Absolute Correlation: A correlation of 0.0420, while winning the competition, is still very low in absolute terms. Reconstructed speech is not yet "audible" or high-fidelity.
  • Feature Ambiguity: It remains unclear if the model is truly reconstructing specific frequency-based features of the mel spectrogram or if it is primarily relying on the shared features between the envelope and the spectrogram.
  • Data Scarcity: To move from 0.04 to a level usable in consumer devices, much larger datasets and even more sophisticated non-linear models will be required.

Paper Citation: Xiran Xu et al., "ConvConcatNet: A Deep Convolutional Neural Network to Reconstruct Mel Spectrogram from the EEG," ICASSP 2024. Link to Code

Summary updated

Abstract

To investigate the processing of speech in the brain, simple linear models are commonly used to establish a relationship between brain signals and speech features. However, these linear models are ill-equipped to model a highly dynamic and complex non-linear system like the brain. Although non-linear methods with neural networks have been developed recently, reconstructing unseen stimuli from unseen subjects' EEG is still a highly challenging task. This work presents a novel method, ConvConcatNet, to reconstruct mel-specgrams from EEG, in which the deep convolution neural network and extensive concatenation operation were combined. With our ConvConcatNet model, the Pearson correlation between the reconstructed and the target mel-spectrogram can achieve 0.0420, which was ranked as No.1 in the Task 2 of the Auditory EEG Challenge. The codes and models to implement our work will be available on Github: https://github.com/xuxiran/ConvConcatNet
BibTeX
@misc{xu2024convconcatnet,
  author = {Xiran Xu and Bo Wang and Yujie Yan and Haolin Zhu and Zechen Zhang and Xihong Wu and Jing Chen},
  title = {{ConvConcatNet: a deep convolutional neural network to reconstruct mel spectrogram from the EEG}},
  year = {2024},
  eprint = {2401.04965},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.04965},
}
Figure