Abstract
Extracting stimulus-related signals from electroencephalography (EEG) data is challenging due to a low signal-to-noise ratio (SNR) and inter-subject variability. We present Deep Multiway Canonical Correlation Analysis (DMCCA), a deep learning framework that normalizes EEG data across subjects by maximizing inter-subject correlation while minimizing reconstruction error through an auto-encoder architecture. Evaluated on natural speech and music datasets, DMCCA significantly outperforms traditional linear methods, achieving up to a 0.29 increase in Pearson correlation and enhancing the extraction of shared neural responses to naturalistic stimuli.
Introduction
Electroencephalography (EEG) provides high temporal resolution for monitoring brain activity, yet stimulus-related signals are often obscured by significant noise, resulting in an average SNR of approximately -20 dB. Traditional methods rely on repeated trials of short stimuli to average out noise, a technique that is impractical for naturalistic stimuli like continuous speech or music. Furthermore, individual differences in brain anatomy and functional organization necessitate "normalization" to find common neural representations across subjects. While linear Multiway Canonical Correlation Analysis (MCCA) is a standard tool for this task, it often fails to capture the complex, non-linear relationships inherent in auditory processing.
Methods: Deep Multiway CCA (DMCCA)
The proposed DMCCA architecture extends multi-view learning by incorporating a deep auto-encoder structure. The model consists of three primary components:
- Shared Encoders: Each subject's EEG data ($x_n$) is processed by an individual neural network encoder ($f_n$) that projects high-dimensional data into a lower-dimensional common subspace.
- Correlation Maximization: The objective function maximizes the correlation between these projected representations across all $N$ subjects, forcing the network to isolate stimulus-driven activity from subject-specific noise.
- Reconstruction Regularization: Unlike previous deep CCA variants, DMCCA includes decoders that attempt to reconstruct the original EEG signals from the shared representation ($y$). This reconstruction task acts as a critical regularizer, preventing the model from overfitting to spurious correlations in the noisy data.
The joint loss function optimizes a weighted combination of inter-subject correlation and Mean Squared Error (MSE) of the reconstruction.
Experimental Results
The model was validated using two datasets: a speech dataset (6 subjects) and the NMED-H music dataset (48 subjects). DMCCA demonstrated superior performance over linear MCCA (LMCCA) across all metrics.
Correlation Improvements
DMCCA achieved substantial gains in Pearson correlation values, with an average improvement of 0.08 for speech and 0.29 for music. As shown in the subject-wise analysis, DMCCA consistently provided higher correlation scores than the linear baseline, indicating a more effective extraction of the shared stimulus response.
Classification and Robustness
In "match-vs-mismatch" classification tasks, DMCCA outperformed linear methods across various segment lengths (1s to 32s). For instance, on the NMED-H dataset with 4s segments, DMCCA reached significantly higher performance metrics compared to LMCCA. The inclusion of the decoder was found to be essential; without the reconstruction loss, the deep network tended to overfit the high-dimensional noise inherent in EEG recordings.
Discussion and Conclusion
The success of DMCCA highlights that the brain's response to complex, naturalistic audio is fundamentally non-linear. By leveraging deep learning and auto-encoder regularization, DMCCA effectively "normalizes" EEG data across subjects, significantly boosting the usable signal. While computationally more intensive than linear methods, the framework offers a robust path forward for improving brain-computer interfaces (BCIs) and auditory tracking technologies. Future research will focus on scaling the model to larger cohorts and improving the interpretability of the learned non-linear features.
Summary updated