Hybrid Brain-Computer Interfaces: Improving Mental Task Classification Performance through Fusion of Neurophysiological Modalities.

Hubert BanvilleEspaceINRS 2015

Abstract

This research demonstrates the efficacy of a multimodal neuroimaging approach for classifying mental states within a Brain-Computer Interface (BCI) framework. By fusing Electroencephalography (EEG) and functional Near-Infrared Spectroscopy (fNIRS), we developed a hybrid BCI (hBCI) to improve binary classification performance across seven distinct mental tasks. Data from nine participants revealed that while EEG provides rapid detection, the addition of fNIRS significantly enhances accuracy during sustained cognitive activity. The multimodal approach yielded an average accuracy increase of 8.5% during the final five seconds of task performance. Furthermore, the study identifies specific neurophysiological markers correlated with subjective user experience, providing a foundation for more flexible and user-aware BCI systems.

Introduction

Brain-Computer Interfaces (BCIs) serve as a critical communication bridge for individuals with severe motor impairments, translating neural activity directly into external commands. Historically, most non-invasive BCIs have relied on Electroencephalography (EEG) due to its high temporal resolution. However, EEG-based systems often suffer from low spatial resolution and significant inter-user variability, leading to a phenomenon known as "BCI-illiteracy," where a subset of users cannot achieve the 70% accuracy threshold required for reliable control.

To address these limitations, this study investigates a hybrid BCI (hBCI) paradigm that integrates EEG with functional Near-Infrared Spectroscopy (fNIRS). While EEG measures the brain's electrical activity, fNIRS tracks hemodynamic changes (blood oxygenation levels) similar to fMRI but in a portable format. By combining these complementary modalities, we aim to leverage the speed of EEG and the spatial reliability of fNIRS to create a more robust classification system.

Methodology

Experimental Design and Tasks

The study involved 12 healthy participants (9 of whom provided valid datasets after noise filtering) performing seven mental tasks across three recording sessions. These tasks were selected to recruit diverse neural mechanisms:

  1. Mental Rotation (ROT): Imagining the rotation of 3D objects.
  2. Word Generation (WORD): Thinking of words starting with a specific letter.
  3. Mental Subtraction (SUB): Successive subtractions of a small number from a large one.
  4. Mental Singing (SING): Mentally performing a familiar song.
  5. Mental Navigation (NAV): Imagining walking through a familiar environment.
  6. Motor Imagery (MI): Imagining rhythmic clenching of the right hand.
  7. Face Imagery (FACE): Imagining the face of a familiar person.

Classification and Fusion

We employed $l_1$-regularized Support Vector Machines (SVMs) for binary classification. The $l_1$ penalty was specifically chosen for its sparsity-inducing properties, allowing the model to automatically select the most relevant features from a high-dimensional dataset containing thousands of EEG and fNIRS variables. We compared three conditions: EEG-only, fNIRS-only, and a feature-level fusion where both modalities were combined into a single feature vector.

Subjective Assessment

Following each task, participants rated their experience using the NASA Task Load Index (TLX). This allowed for a correlation analysis between neurophysiological features and subjective dimensions such as mental effort, frustration, and perceived performance.

Key Results

Performance Gains through Fusion

The multimodal EEG-fNIRS approach consistently outperformed single-modality systems. When using short, one-second windows, the fusion led to a modest average increase in Cohen’s kappa of 0.03 (approximately 1.5% accuracy). However, the benefit of fusion became much more pronounced during sustained activity. In the final five seconds of the 15-second trials, the hybrid system increased classification accuracy by an average of 8.5% compared to EEG alone.

Optimal Task Pairings

The study identified that the most discriminable task pairs typically combined a high-effort cognitive task with a passive imagery task. The pairing of Mental Rotation vs. Motor Imagery emerged as the most effective for classification, likely due to the distinct and non-overlapping neural regions recruited by these two activities.

Feature Importance

In the EEG modality, the Alpha and Beta frequency bands provided the most discriminative information. For fNIRS, Deoxy-hemoglobin (HbR) levels were generally more useful for classification than Oxy-hemoglobin (HbO), likely due to HbR's higher spatial specificity and lower susceptibility to systemic physiological noise.

Discussion

Temporal Synergy

The success of the hBCI is rooted in the different timescales of the two modalities. EEG signals peak almost instantly (within 3-4 seconds), providing a "quick start" for the BCI. In contrast, the fNIRS hemodynamic response is slower, peaking around 11 seconds. The fusion captures both the initial electrical burst and the sustained metabolic "steady burn," which is particularly useful for maintaining accuracy in long-duration mental tasks.

Addressing BCI-Illiteracy

A significant finding was that fNIRS often provided critical information for participants who performed poorly with EEG alone. This suggests that hybrid systems can be more inclusive, offering a viable control method for users whose electrical brain patterns are difficult to decode.

Predicting User State

The correlation analysis revealed specific neurophysiological markers that align with user frustration and mental load. This opens the door for "user-aware" BCIs that can dynamically adjust their difficulty or interface based on the user's real-time cognitive state, potentially reducing frustration and improving long-term usability.

Limitations and Future Directions

Despite the performance gains, challenges remain. The inherent delay in the fNIRS hemodynamic response (the 11-second peak) is a bottleneck for applications requiring near-instantaneous commands, such as wheelchair navigation. Additionally, the complexity of wearing a dual-sensor cap may be cumbersome for daily use. Future research should focus on developing integrated hardware and exploring "inter-subject" classification models that work with minimal calibration for new users.

Summary updated

Abstract

Ce projet démontre l’utilisation de multiples techniques de neuroimagerie pour la classification d’états
\nmentaux dans le contexte d’une interface cerveau-ordinateur (ICO). Basé sur de récentes études en
\nélectroencéphalographie (EEG) et en spectroscopie proche-infrarouge (SPIR) ayant démontré que des
\ntâches telles que l’imagerie motrice et l’arithmétique mentale induisent des patrons d’activité neurophysiologique
\nparticuliers, un paradigme d’ICO hybride (ICOh) combinant l’EEG et la SPIR est développé
\nafin d’améliorer la performance de classification binaire d’un tel système.
\nDes signaux d’EEG et de SPIR ont été enregistrés sur neuf participants réalisant sept tâches mentales
\n(génération de mots, rotation mentale, soustraction mentale, chant mental, navigation mentale, imagerie
\nmotrice et imagerie de visages). Ensuite, des machines à vecteurs de support avec régularisation l1
\nont été entraînées sur chacune des paires de tâches possibles 1) en EEG, 2) en SPIR, et 3) en EEG
\net en SPIR conjointement, afin d’identifier les meilleures tâches et d’évaluer l’utilité d’une approche
\nmultimodale. L’évaluation subjective de la charge mentale induite par chaque tâche a aussi été collectée
\nafin d’en estimer la facilité d’utilisation et d’identifier des traits caractéristiques neurophysiologiques qui
\npermettraient de prédire l’expérience subjective d’un utilisateur à travers une analyse des corrélations
\nentre traits caractéristiques et dimensions subjectives.
\nL’approche multimodale EEG-SPIR a permis d’améliorer la valeur maximale du kappa de Cohen
\n(mesure de l’accord entre deux observateurs) de 0.03 lorsque basée sur des fenêtres d’une seconde, ce
\nqui correspond à une augmentation de l’exactitude de classification de 1.5% pour des classes équilibrées.
\nCette augmentation s’est révélée encore plus importante (variation de 0.17, soit 8.5% en exactitude)
\nlorsque les cinq dernières secondes des tâches étaient prises en compte. L’analyse de l’EEG et de la SPIR
\na de plus permis de révéler l’importance relative de différentes régions du cerveau et de types de traits
\ncaractéristiques, alors que l’analyse des corrélations a permis d’identifier les traits caractéristiques les plus
\nprometteurs pour prédire l’expérience subjective d’un utilisateur, bien qu’une évaluation plus approfondie
\nsera nécessaire pour valider ces résultats. Ces travaux fournissent une base solide pour d’autres études
\nsur les ICOh cherchant à améliorer la performance de classification vers la réalisation d’ICO plus efficaces
\net flexibles. <br /><br />The symbols and special characters used in the original abstract could not be transcribed due 
\nto technical problems. Please use the PDF version to read the abstract. this work, we demonstrate the use of multiple neuroimaging modalities to classify brain states in
\na Brain-Computer Interface. Based on recent electroencephalography (EEG) and near-infrared spectroscopy
\n(NIRS) studies that showed that tasks such as motor imagery and mental arithmetic induce
\nspecific neural response patterns, we propose a hybrid BCI (hBCI) paradigm in which EEG and NIRS
\ndata are fused to improve binary classification performance.
\nWe recorded full-head EEG and partial-head NIRS data from nine participants performing seven
\nmental tasks (word generation, mental rotation, subtraction, singing and navigation, motor and face
\nimagery). l1-Support Vector Machines were trained for each possible pair of tasks on 1) EEG features,
\n2) NIRS features, and 3) EEG and NIRS features together, to identify the best pairs and assess the
\nusefulness of a multimodal approach. Subjective ratings were also collected to assess the usability of
\neach task and carry out a correlation analysis to discover features that could help predict the subjective
\nexperience of a user.
\nThe EEG-NIRS multimodal approach led to an average increase in peak kappa (an inter-rater agreement
\nmetric) of 0.03 when using features extracted from one-second windows (equivalent to an increase
\nof 1.5% in classification accuracy for balanced classes). The increase was much stronger (change of 0.17,
\ncorresponding to an 8.5% accuracy increase) when focusing on the last five seconds of the 15-second
\ntrials. The EEG and NIRS analyses further unveiled relevant brain regions and important feature types,
\nwhile preliminary results from the correlation analysis uncovered features that could be used to predict
\nthe subjective experience of a BCI user. This work provides a basis for future hBCI studies aiming to
\nimprove classification performance toward more efficient and flexible BCIs.
BibTeX
@inproceedings{banville2015hybrid,
  author = {Hubert Banville},
  title = {{Hybrid Brain-Computer Interfaces: Improving Mental Task Classification Performance through Fusion of Neurophysiological Modalities.}},
  booktitle = {EspaceINRS},
  year = {2015},
}
Figure