Abstract
This research demonstrates the efficacy of a multimodal neuroimaging approach for classifying mental states within a Brain-Computer Interface (BCI) framework. By fusing Electroencephalography (EEG) and functional Near-Infrared Spectroscopy (fNIRS), we developed a hybrid BCI (hBCI) to improve binary classification performance across seven distinct mental tasks. Data from nine participants revealed that while EEG provides rapid detection, the addition of fNIRS significantly enhances accuracy during sustained cognitive activity. The multimodal approach yielded an average accuracy increase of 8.5% during the final five seconds of task performance. Furthermore, the study identifies specific neurophysiological markers correlated with subjective user experience, providing a foundation for more flexible and user-aware BCI systems.
Introduction
Brain-Computer Interfaces (BCIs) serve as a critical communication bridge for individuals with severe motor impairments, translating neural activity directly into external commands. Historically, most non-invasive BCIs have relied on Electroencephalography (EEG) due to its high temporal resolution. However, EEG-based systems often suffer from low spatial resolution and significant inter-user variability, leading to a phenomenon known as "BCI-illiteracy," where a subset of users cannot achieve the 70% accuracy threshold required for reliable control.
To address these limitations, this study investigates a hybrid BCI (hBCI) paradigm that integrates EEG with functional Near-Infrared Spectroscopy (fNIRS). While EEG measures the brain's electrical activity, fNIRS tracks hemodynamic changes (blood oxygenation levels) similar to fMRI but in a portable format. By combining these complementary modalities, we aim to leverage the speed of EEG and the spatial reliability of fNIRS to create a more robust classification system.
Methodology
Experimental Design and Tasks
The study involved 12 healthy participants (9 of whom provided valid datasets after noise filtering) performing seven mental tasks across three recording sessions. These tasks were selected to recruit diverse neural mechanisms:
- Mental Rotation (ROT): Imagining the rotation of 3D objects.
- Word Generation (WORD): Thinking of words starting with a specific letter.
- Mental Subtraction (SUB): Successive subtractions of a small number from a large one.
- Mental Singing (SING): Mentally performing a familiar song.
- Mental Navigation (NAV): Imagining walking through a familiar environment.
- Motor Imagery (MI): Imagining rhythmic clenching of the right hand.
- Face Imagery (FACE): Imagining the face of a familiar person.
Classification and Fusion
We employed $l_1$-regularized Support Vector Machines (SVMs) for binary classification. The $l_1$ penalty was specifically chosen for its sparsity-inducing properties, allowing the model to automatically select the most relevant features from a high-dimensional dataset containing thousands of EEG and fNIRS variables. We compared three conditions: EEG-only, fNIRS-only, and a feature-level fusion where both modalities were combined into a single feature vector.
Subjective Assessment
Following each task, participants rated their experience using the NASA Task Load Index (TLX). This allowed for a correlation analysis between neurophysiological features and subjective dimensions such as mental effort, frustration, and perceived performance.
Key Results
Performance Gains through Fusion
The multimodal EEG-fNIRS approach consistently outperformed single-modality systems. When using short, one-second windows, the fusion led to a modest average increase in Cohen’s kappa of 0.03 (approximately 1.5% accuracy). However, the benefit of fusion became much more pronounced during sustained activity. In the final five seconds of the 15-second trials, the hybrid system increased classification accuracy by an average of 8.5% compared to EEG alone.
Optimal Task Pairings
The study identified that the most discriminable task pairs typically combined a high-effort cognitive task with a passive imagery task. The pairing of Mental Rotation vs. Motor Imagery emerged as the most effective for classification, likely due to the distinct and non-overlapping neural regions recruited by these two activities.
Feature Importance
In the EEG modality, the Alpha and Beta frequency bands provided the most discriminative information. For fNIRS, Deoxy-hemoglobin (HbR) levels were generally more useful for classification than Oxy-hemoglobin (HbO), likely due to HbR's higher spatial specificity and lower susceptibility to systemic physiological noise.
Discussion
Temporal Synergy
The success of the hBCI is rooted in the different timescales of the two modalities. EEG signals peak almost instantly (within 3-4 seconds), providing a "quick start" for the BCI. In contrast, the fNIRS hemodynamic response is slower, peaking around 11 seconds. The fusion captures both the initial electrical burst and the sustained metabolic "steady burn," which is particularly useful for maintaining accuracy in long-duration mental tasks.
Addressing BCI-Illiteracy
A significant finding was that fNIRS often provided critical information for participants who performed poorly with EEG alone. This suggests that hybrid systems can be more inclusive, offering a viable control method for users whose electrical brain patterns are difficult to decode.
Predicting User State
The correlation analysis revealed specific neurophysiological markers that align with user frustration and mental load. This opens the door for "user-aware" BCIs that can dynamically adjust their difficulty or interface based on the user's real-time cognitive state, potentially reducing frustration and improving long-term usability.
Limitations and Future Directions
Despite the performance gains, challenges remain. The inherent delay in the fNIRS hemodynamic response (the 11-second peak) is a bottleneck for applications requiring near-instantaneous commands, such as wheelchair navigation. Additionally, the complexity of wearing a dual-sensor cap may be cumbersome for daily use. Future research should focus on developing integrated hardware and exploring "inter-subject" classification models that work with minimal calibration for new users.
Summary updated