Mental Task Evaluation for Hybrid NIRS-EEG Brain-Computer Interfaces

Hubert J. Banville, Rishabh Gupta, Tiago H. FalkComput. Intell. Neurosci 2017 · DOI · OpenReview

TL;DR

This study explores combining EEG and NIRS brain signals to improve the accuracy of brain-computer interfaces (BCIs) that rely on mental tasks. By fusing data from both modalities, they achieved a significant boost in classification performance, particularly when focusing on optimal NIRS signal windows, suggesting a more robust and efficient BCI design.

What Problem Does This Paper Address?

Brain-computer interfaces (BCIs) offer a way to communicate and control devices using brain signals, bypassing traditional motor pathways. While EEG-based BCIs are common, they can be limited by individual variability and the need for specific brain regions to be active. Other neuroimaging techniques like NIRS offer complementary information, but their integration into hybrid BCIs (hBCIs) to improve classification accuracy for mental tasks remains an active area of research. This paper aims to address this by systematically evaluating the performance of different mental tasks and assessing the benefits of fusing EEG and NIRS data for more effective BCI control. The challenge lies in identifying optimal mental tasks and understanding how combining these two modalities can overcome the limitations of each individual one.

Methods

The researchers employed a hybrid BCI approach, simultaneously recording EEG and NIRS data from participants performing seven distinct mental tasks. These tasks were categorized as "brain teasers" (mental rotation, word generation, mental subtraction), "dynamic imagery" (mental singing, mental navigation, motor imagery), and "static imagery" (face imagery).

Key methodological aspects include:

  • Simultaneous NIRS-EEG Recording: Data was collected concurrently from 62 EEG channels and 60 NIRS channels.
  • Mental Task Paradigm: Participants performed tasks covertly for 15-second intervals, interspersed with rest periods.
  • Feature Extraction: EEG data was analyzed using band power features across various frequency bands, while NIRS data focused on changes in oxygenated and deoxygenated hemoglobin. Features were extracted from one-second sliding windows.
  • Classification: A linear kernel Support Vector Machine (SVM) with an L1 penalty was used for binary classification between pairs of tasks. This approach helps in feature selection by promoting sparsity.
  • Performance Evaluation: Cohen's kappa (κ) was used as the primary metric for classification performance, with a focus on peak performance across time and subjects.
  • Fusion Strategy: Features from EEG and NIRS were concatenated and fed into the SVM classifier to assess the impact of multimodal integration.
  • Feature Importance Analysis: SVM weights were used to identify the most informative EEG and NIRS features and their corresponding brain regions.

Datasets

The study utilized data collected from nine healthy participants (five female, three left-handed, mean age 24.6 years) who completed three recording sessions. No new benchmark datasets were introduced; instead, the researchers collected novel NIRS-EEG data specifically for this study.

Key Insights and Ideas

  • Task Pair Synergy: Combinations of "brain teaser" tasks (like mental rotation and subtraction) with "imagery" tasks (like motor imagery) consistently yielded the best classification performance across all modalities.
  • NIRS's Temporal Advantage: While EEG showed peak performance earlier in the task execution (around 4 seconds), NIRS data became more informative later (around 11 seconds), likely due to the hemodynamic response delay.
  • Fusion's Power in Optimal Windows: The most significant performance gains from fusing EEG and NIRS were observed when focusing on time windows where NIRS signals were particularly strong, suggesting a strategic approach to data fusion.
  • Complementary Modalities: The study demonstrated that EEG and NIRS provide complementary information, with one modality often excelling where the other was weaker, making their combination particularly beneficial.
  • Individualized BCI Potential: The results suggest that hBCIs could be particularly valuable for individuals with "BCI illiteracy" (poor performance with unimodal systems), as fusion can compensate for weaknesses in one modality.
  • Feature Importance Reveals Brain Activity: The analysis of SVM weights provided insights into which brain regions and feature types were most critical for discriminating between different mental tasks, aligning with known neurophysiology.

Key Results

  • EEG-Only Performance: Achieved an average peak kappa (κ) of 0.63, with some task pairs reaching high performance (κ > 0.7).
  • NIRS-Only Performance: Achieved an average peak kappa (κ) of 0.47, with a longer peak time (11.5 seconds) compared to EEG.
  • EEG-NIRS Fusion Performance:
    • An average increase of 0.03 in peak kappa was observed when using features from one-second windows across the entire task (equivalent to a 1.5% accuracy increase).
  • A much stronger average increase of 0.20 in kappa was achieved when focusing on time windows of high NIRS performance (equivalent to a 10% accuracy increase).
  • Best Performing Task Pairs: Combinations of brain teasers (ROT, SUB, WORD) and imagery tasks (MI, FACE, SING) were consistently the top performers. ROT-MI was a particularly strong pair.
  • Worst Performing Task Pairs: Pairs involving static imagery tasks like FACE-MI and SING-FACE generally performed poorly.
  • Feature Importance: EEG features related to alpha and beta bands were highly important, particularly from occipital and parietal regions. NIRS features related to HbR and HbO were also significant, with prefrontal and frontal regions being informative.

Limitations and Future Work

  • Real-time Implementation Challenges: The study acknowledges that translating these findings to a real-time BCI application will require overcoming challenges related to setup time, computational efficiency, and robustness to noise.
  • NIRS Hemodynamic Delay: The significant delay in NIRS signals (peaking around 11 seconds) makes it less suitable for rapid BCI control on its own, but its fusion with EEG mitigates this by providing earlier information.
  • Task Difficulty and Subjectivity: The paper notes that tasks like mental subtraction and face imagery were perceived as demanding and frustrating, suggesting that optimizing task difficulty and selection is crucial for user experience and performance.
  • Future Research Directions: The authors emphasize the need for online BCI implementation using the identified optimal task pairs. Further research could explore different feature extraction methods, window sizes, and fusion strategies to maximize performance and usability. Investigating the impact of task difficulty adjustments and individual differences in more detail is also recommended.

Summary updated

Abstract

Based on recent electroencephalography (EEG) and near-infrared spectroscopy (NIRS) studies that showed that tasks such as motor imagery and mental arithmetic induce specific neural response patterns, we propose a hybrid brain-computer interface (hBCI) paradigm in which EEG and NIRS data are fused to improve binary classification performance. We recorded simultaneous NIRS-EEG data from nine participants performing seven mental tasks (word generation, mental rotation, subtraction, singing and navigation, and motor and face imagery). Classifiers were trained for each possible pair of tasks using (<svg xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.w3.org/2000/svg" style="vertical-align:-0.04981995pt" id="M1" height="8.39864pt" version="1.1" viewBox="-0.0498162 -8.34882 6.36303 8.39864" width="6.36303pt"><g transform="matrix(.013,0,0,-0.013,0,0)"><path id="g113-50" d="M384 0V27C293 34 287 42 287 114V635C232 613 172 594 109 583V559L157 557C201 555 205 550 205 499V114C205 42 199 34 109 27V0H384Z"/></g></svg>) EEG features alone, (<svg xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.w3.org/2000/svg" style="vertical-align:-0.04981995pt" id="M2" height="8.39864pt" version="1.1" viewBox="-0.0498162 -8.34882 6.36303 8.39864" width="6.36303pt"><g transform="matrix(.013,0,0,-0.013,0,0)"><path id="g113-51" d="M412 140C382 77 369 73 315 73H129L270 222C362 320 402 379 402 466C402 571 322 635 234 635C177 635 130 609 99 576L42 495L64 475C90 514 133 568 201 568C274 568 318 519 318 435C318 349 255 267 193 193C144 135 87 78 32 23V0H405C417 45 427 89 440 131L412 140Z"/></g></svg>) NIRS features alone, and (<svg xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.w3.org/2000/svg" style="vertical-align:-0.2063904pt" id="M3" height="8.55521pt" version="1.1" viewBox="-0.0498162 -8.34882 6.36303 8.55521" width="6.36303pt"><g transform="matrix(.013,0,0,-0.013,0,0)"><path id="g113-52" d="M285 378C315 398 338 416 353 432C373 451 384 474 384 503C384 579 325 635 236 635H235C182 635 136 610 108 579L65 516L85 496C110 533 150 575 205 575C258 575 300 543 300 481C300 407 232 369 141 339L147 310C163 315 188 321 211 321C268 321 338 284 338 192C338 94 288 40 217 40C160 40 119 68 93 91C85 98 77 97 69 91C60 84 47 71 46 58C44 46 48 35 62 22C75 10 116 -12 162 -12C234 -12 424 62 424 224C424 297 373 359 285 376V378Z"/></g></svg>) EEG and NIRS features combined, to identify the best task pairs and assess the usefulness of a multimodal approach. The NIRS-EEG approach led to an average increase in peak kappa of 0.03 when using features extracted from one-second windows (equivalent to an increase of 1.5% in classification accuracy for balanced classes). The increase was much stronger (0.20, corresponding to an 10% accuracy increase) when focusing on time windows of high NIRS performance. The EEG and NIRS analyses further unveiled relevant brain regions and important feature types. This work provides a basis for future NIRS-EEG hBCI studies aiming to improve classification performance toward more efficient and flexible BCIs.
BibTeX
@article{banville2017mental,
  author = {Hubert J. Banville and Rishabh Gupta and Tiago H. Falk},
  title = {{Mental Task Evaluation for Hybrid NIRS-EEG Brain-Computer Interfaces}},
  journal = {Comput. Intell. Neurosci.},
  year = {2017},
  doi = {10.1155/2017/3524208},
}
Figure