EF-Net: Mental State Recognition by Analyzing Multimodal EEG-fNIRS via CNN

Xiang Zhang, Aniqa Arif, Yihe Wang, Rui Yin, Xiang Zhang 0012, Ahmed HelmySensors 2024 · DOI

TL;DR

This paper introduces EF-Net, a novel deep learning model that effectively combines EEG and fNIRS brain signals to recognize mental states. EF-Net demonstrates superior performance, particularly in subject-independent settings, paving the way for more robust and generalizable brain-computer interfaces.

What Problem Does This Paper Address?

Accurately recognizing human mental states is a critical challenge in neuroscience and has significant implications for brain-computer interfaces (BCIs), mental health monitoring, and understanding cognitive processes. While Electroencephalography (EEG) and functional Near-Infrared Spectroscopy (fNIRS) are promising non-invasive methods for capturing brain activity, their combined potential remains underexplored, especially in real-world scenarios where models need to generalize to unseen individuals. Existing research often focuses on subject-dependent or semi-dependent analyses, which limit applicability. The key challenge is to develop models that can effectively learn from multimodal brain data and generalize across diverse subjects, overcoming individual variations and noise.

Methods

The core contribution of this paper is the EF-Net model, a Convolutional Neural Network (CNN)-based architecture designed for multimodal EEG-fNIRS analysis. EF-Net features two specialized branches:

  • EEG Modality Branch: This branch focuses on extracting temporal features from EEG signals. It employs multiple 2D convolutional blocks with specific kernel sizes to scan temporal patterns across channels. Max pooling, dropout, and batch normalization are used to refine feature extraction and prevent overfitting.
  • fNIRS Modality Branch: This branch is designed to capture spatial features from fNIRS data. It uses fewer convolutional blocks due to the lower temporal resolution of fNIRS but incorporates kernels that capture both temporal and spatial aspects. Similar regularization techniques are applied.

These two branches independently learn representations, which are then concatenated and fed into fully connected layers for final classification. The paper also explores single-modality performance by deactivating one branch.

Datasets

The study utilizes the EEG-fNIRS WG dataset from [17]. This is an open-access dataset that provides simultaneous EEG and fNIRS recordings. Specifically, Dataset C was used, which includes 26 subjects, each participating in 60 trials (10 Word Generation and 10 Baseline) across three sessions. The dataset is balanced between the two classes.

  • fNIRS: 72 channels, 10 Hz sampling rate.
  • EEG: 30 channels, 1000 Hz sampling rate.

The paper details preprocessing steps, including downsampling, filtering, artifact removal, and segmenting data into 5-second windows with 1-second overlap, resulting in 9360 samples per modality per subject.

Key Insights and Ideas

  • Multimodal Fusion is Key: The study strongly suggests that combining EEG and fNIRS data yields superior results, especially in the challenging subject-independent setting. This highlights the complementary nature of the temporal (EEG) and spatial (fNIRS) information.
  • Subject-Independent Performance Matters: The paper emphasizes the importance of subject-independent analysis for real-world applications. While subject-dependent models perform better, their limited generalizability makes them less practical for widespread use.
  • CNNs are Effective for Multimodal Brain Data: EF-Net's success demonstrates the power of CNNs in learning complex representations from both EEG and fNIRS signals without extensive manual feature engineering.
  • Careful Preprocessing is Crucial: The detailed preprocessing steps, including windowing and downsampling, are vital for preparing the data for the deep learning model and achieving optimal performance.

Key Results

EF-Net achieved state-of-the-art performance across various settings:

  • Subject-Dependent: EF-Net achieved an F1 score of 99.36%, surpassing the best baseline by 1.83%.
  • Subject-Semidependent: EF-Net achieved an F1 score of 98.31%, surpassing the best baseline by 4.34%.
  • Subject-Independent: EF-Net achieved an F1 score of 65.05%, surpassing the best baseline by 2.13%.

The paper also reports that the combined use of EEG and fNIRS data consistently outperformed single modalities, particularly in the subject-independent setting.

Limitations and Future Work

The authors acknowledge that the performance in the subject-independent setting, while improved, still has room for enhancement. They suggest several future directions:

  • Extending EF-Net: Applying the model to a broader range of EEG-fNIRS multimodal brain activity learning tasks beyond mental state recognition.
  • Exploring Advanced Architectures: Investigating state-of-the-art model architectures like transformer-based models for multimodal brain activity learning.
  • Improving Subject-Independent Results: Exploring methods to further improve subject-independent performance, such as outlier removal from the dataset.
  • Integration with Other Modalities: Combining EEG-fNIRS with other data sources like eye-tracking for enhanced applications in VR and assistive technologies.

Summary updated

BibTeX
@article{zhang2024ef,
  author = {Xiang Zhang and Aniqa Arif and Yihe Wang and Rui Yin and Xiang Zhang and Ahmed Helmy},
  title = {{EF-Net: Mental State Recognition by Analyzing Multimodal EEG-fNIRS via CNN}},
  journal = {Sensors},
  year = {2024},
  doi = {10.3390/s24061889},
}
Figure