Multi-Modal Electrophysiological Source Imaging With Attention Neural Networks Based on Deep Fusion of EEG and MEG

Meng Jiao, Shihao Yang, Xiaochen Xian, Neel Fotedar, Feng LiuIEEE Transactions on Neural Systems and Rehabilitation Engineering 2024 · DOI

TL;DR

This paper introduces MMDF-ANN, a novel deep learning framework that fuses EEG and MEG data for more accurate brain source localization. By leveraging the complementary information from both modalities and employing an attention mechanism, MMDF-ANN significantly improves upon existing methods, especially for complex and noisy brain activity.

What Problem Does This Paper Address?

Electrophysiological Source Imaging (ESI) aims to pinpoint the brain regions generating electrical signals detected by EEG and MEG. While EEG and MEG offer complementary information (MEG is better for tangential sources, EEG for radial), most current deep learning approaches for ESI only use one modality. This leaves the potential benefits of combining EEG and MEG largely untapped. The challenge lies in effectively fusing these two distinct data streams within a deep learning framework to achieve superior source localization accuracy.

Methods

The proposed Multi-Modal Deep Fusion (MMDF) framework using Attention Neural Networks (MMDF-ANN) tackles this by:

  • Dual CNN Feature Extraction: Two separate Convolutional Neural Network (CNN) modules are used to independently process EEG and MEG data. This allows for the extraction of modality-specific features.
  • Dilated Convolutions: To capture broader spatial dependencies without increasing network depth, dilated convolutions are employed. This helps the CNNs "see" more of the spatial context.
  • Channel-wise Attention Mechanism: An attention module is introduced to adaptively weigh the importance of features extracted from each modality. This allows the network to dynamically focus on the most relevant information from EEG and MEG at different stages.
  • Early Fusion: The fusion of EEG and MEG information happens at the feature level (early fusion) rather than at the decision level, which is argued to be more principled and effective.
  • Topological Loss Function: Beyond the standard Mean Squared Error (MSE), a novel topological loss is introduced. This loss penalizes source estimates based on their geodesic distance on the brain's surface, encouraging more accurate localization, especially for extended sources.

Datasets

  • Synthetic Data: Generated to simulate various brain source activation patterns, including different levels of "neighborhoods" (representing source extent) and signal-to-noise ratios (SNRs).
  • Real Data:
    • Face Perception Dataset: From the SPM database, containing simultaneous EEG and MEG recordings from a subject performing a face perception task.
  • Epilepsy Dataset: From the Epilepsy Center Freiburg, Germany, containing high-resolution EEG data from a patient with focal epilepsy. (Note: MEG data was not available for this specific dataset, so only EEG was used for MMDF-ANN in this case).

Key Insights and Ideas

  • Complementarity is Key: Combining EEG and MEG data demonstrably improves source localization accuracy compared to using either modality alone. This highlights the value of multimodal approaches.
  • Attention for Fusion: An attention mechanism is crucial for effectively integrating information from different modalities, allowing the network to learn which modality is more informative for specific aspects of the signal.
  • Topological Loss for Accuracy: The novel topological loss function is a significant contribution, providing better guidance for model training by considering the spatial relationships between brain regions, leading to more precise localization.
  • Deep Learning Outperforms Traditional Methods: The proposed deep learning framework, especially for extended sources and low SNR conditions, shows superior performance compared to traditional algorithms like MNE, sLORETA, and dSPM.

Key Results

  • Improved Accuracy: MMDF-ANN consistently achieved higher Area Under the Precision-Recall Curve (AUPRC) and lower Localization Error (LE) compared to benchmark algorithms across various synthetic data scenarios (single source, multiple sources, varying SNRs, and source extents).
  • Robustness: The framework demonstrated good stability in reconstructing sources with extended activation areas and in low signal-to-noise ratio (SNR) conditions.
  • Real-World Performance: On real datasets (face perception and epilepsy), MMDF-ANN provided more precise and compact source reconstructions, outperforming traditional methods and even other deep learning approaches like ConvDip. For instance, in the face perception task, MMDF-ANN provided a cleaner estimation of the visual area compared to other methods. In the epilepsy dataset, it offered a cleaner reconstruction of the epileptogenic source activation.
  • Ablation Studies: Confirmed the benefits of multimodal fusion (EEG+MEG vs. single modality), the effectiveness of dilated convolutions, and the positive impact of the topological loss function.

Limitations and Future Work

  • MEG Channel Count: The paper notes that MEG generally outperforms EEG in single-modality performance due to having more channels and being less affected by skull conductivity. This suggests that further improvements might be possible by increasing EEG channel density or developing more sophisticated EEG processing.
  • Computational Resources: Training deep learning models, especially with multimodal data, requires significant computational resources (e.g., GPUs), which could be a barrier for some researchers.
  • Hyperparameter Sensitivity: While Bayesian optimization was used to tune hyperparameters, the paper notes that the model's performance can be sensitive to these choices, suggesting that robust hyperparameter tuning remains an important aspect.
  • Future Work: The authors suggest that their framework could be extended to other electrophysiological signals and applied to a wider range of neuroscience and clinical applications. Further exploration of attention mechanisms and fusion strategies could also yield improvements.

Summary updated

Abstract

The process of reconstructing underlying cortical and subcortical electrical activities from Electroencephalography (EEG) or Magnetoencephalography (MEG) recordings is called Electrophysiological Source Imaging (ESI). Given the complementarity between EEG and MEG in measuring radial and tangential cortical sources, combined EEG/MEG is considered beneficial in improving the reconstruction performance of ESI algorithms. Traditional algorithms mainly emphasize incorporating predesigned neurophysiological priors to solve the ESI problem. Deep learning frameworks aim to directly learn the mapping from scalp EEG/MEG measurements to the underlying brain source activities in a data-driven manner, demonstrating superior performance compared to traditional methods. However, most of the existing deep learning approaches for the ESI problem are performed on a single modality of EEG or MEG, meaning the complementarity of these two modalities has not been fully utilized. How to fuse the EEG and MEG in a more principled manner under the deep learning paradigm remains a challenging question. This study develops a Multi-Modal Deep Fusion (MMDF) framework using Attention Neural Networks (ANN) to fully leverage the complementary information between EEG and MEG for solving the ESI inverse problem, which is termed as MMDF-ANN. Specifically, our proposed brain source imaging approach consists of four phases, including feature extraction, weight generation, deep feature fusion, and source mapping. Our experimental results on both synthetic dataset and real dataset demonstrated that using a fusion of EEG and MEG can significantly improve the source localization accuracy compared to using a single-modality of EEG or MEG. Compared to the benchmark algorithms, MMDF-ANN demonstrated good stability when reconstructing sources with extended activation areas and situations of EEG/MEG measurements with a low signal-to-noise ratio.
BibTeX
@article{jiao2024multi,
  author = {Meng Jiao and Shihao Yang and Xiaochen Xian and Neel Fotedar and Feng Liu},
  title = {{Multi-Modal Electrophysiological Source Imaging With Attention Neural Networks Based on Deep Fusion of EEG and MEG}},
  journal = {IEEE Transactions on Neural Systems and Rehabilitation Engineering},
  year = {2024},
  doi = {10.1109/tnsre.2024.3424669},
}
Figure