Abstract
With the rapid development of brain-computer interface (BCI) technology, the effective integration of multimodal biological signals to improve classification accuracy has become a research hotspot. However, existing methods often fail to fully exploit cross-modality correlations in complex cognitive tasks. To address this, this paper proposes a Multi-Branch Convolutional Neural Network with Attention (MBC-ATT) for BCI based cognitive tasks classification. MBC-ATT employs independent branch structures to process electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) signals separately, thereby leveraging the advantages of each modality. To further enhance the fusion of multimodal features, we introduce a cross-modal attention mechanism to discriminate features, strengthening the model's ability to focus on relevant signals and thereby improving classification accuracy. We conducted experiments on the n-back and WG datasets. The results demonstrate that the proposed model outperforms conventional approaches in classification performance, further validating the effectiveness of MBC-ATT in brain-computer interfaces. This study not only provides novel insights for multimodal BCI systems but also holds great potential for various applications.
Introduction
Brain-Computer Interfaces (BCIs) typically rely on non-invasive signals like Electroencephalography (EEG) or functional Near-Infrared Spectroscopy (fNIRS). However, both modalities present inherent trade-offs that limit their individual effectiveness. EEG offers excellent temporal resolution, capturing brain activity in milliseconds, but it is highly susceptible to noise and suffers from poor spatial localization. Conversely, fNIRS provides superior spatial resolution by measuring blood oxygenation changes, but its response is inherently slow, reflecting hemodynamic changes over seconds rather than milliseconds.
While hybrid BCIs aim to combine these signals to overcome individual limitations, traditional fusion methods—such as "early fusion" (stacking raw data) or "late fusion" (averaging decisions)—often fail to capture the complex, dynamic correlations between electrical activity and hemodynamic responses, a relationship known as neurovascular coupling. This paper introduces the Multi-Branch Convolutional Neural Network with Attention (MBC-ATT), a framework designed to intelligently integrate these two distinct data streams by dynamically weighing their contributions.
Methods
The MBC-ATT architecture is designed to leverage the complementary strengths of EEG and fNIRS through a three-stage process:
- Independent Feature Extraction: A dual-branch CNN architecture processes EEG and fNIRS signals separately. The EEG branch utilizes specific kernel sizes (7,1) to capture short-term temporal dependencies, while the fNIRS branch focuses on local fluctuations in blood oxygen levels.
- Cross-Modal Attention Fusion: Moving beyond simple concatenation, the model employs a Query-Key-Value (QKV) interaction. Features from the EEG branch act as a "Query" to probe fNIRS "Keys" and "Values" (and vice versa), allowing the model to dynamically weigh the most relevant information from each modality.
- Multi-Head Design: A 4-head attention mechanism enables the model to analyze cross-modal relationships from multiple "perspectives" simultaneously, enhancing expressive power.
Experimental Design and Datasets
The model was validated using an open-access multimodal dataset (Shin et al., 2018) featuring 26 participants. The experimental paradigm included two distinct cognitive tasks:
- n-back Task: Used to assess working memory load by comparing 0-back, 2-back, and 3-back states.
- Word Generation (WG) Task: A language processing task where participants silently generate words starting with a specific letter, compared against a baseline rest state.
Results and Discussion
The MBC-ATT model demonstrated superior performance across both benchmarks, setting a new bar for cognitive state decoding:
- n-back Task: Achieved 98.13% accuracy, significantly outperforming standard SVM (83.00%) and advanced multimodal DenseNet models (95.10%).
- Word Generation Task: Achieved 98.61% accuracy, a substantial improvement over the traditional SVM baseline (73.99%) and the EF-Net (96.29%).
Statistical analysis using paired t-tests confirmed that these improvements were significant ($p < 0.05$), suggesting the model is robust across different subjects despite the high variability typical of brain signals. Ablation studies further confirmed that the cross-modal attention mechanism is critical; removing it in favor of simple concatenation resulted in a 6% drop in accuracy on the n-back task. This suggests that the dynamic synergy captured by the attention module—allowing the model to "attend" to specific brain regions or time points where the two signals are most complementary—is vital for effective multimodal fusion.
Despite these results, real-world deployment faces challenges. The model requires high-precision temporal synchronization between EEG and fNIRS hardware and entails higher computational costs than simpler architectures. Future work will focus on developing lightweight versions of the model and exploring transfer learning to reduce calibration requirements for new users.
Summary updated