The Study of Influence of Sound on Visual ERP-Based Brain Computer Interface

Guizhi Xu, Yuwei Wu, Mengfan LiSensors 2020 · DOI

TL;DR

This study investigates how auditory stimuli, specifically the speed of an auditory story, impact the performance of visual Brain-Computer Interfaces (BCIs) that rely on Event-Related Potentials (ERPs). The findings reveal that increased auditory speed elevates mental workload, leading to reduced ERP amplitudes, lower accuracy, and decreased information transfer rates in visual BCIs, highlighting the importance of considering environmental factors for real-world BCI applications.

What Problem Does This Paper Address?

Brain-Computer Interfaces (BCIs) that utilize Event-Related Potentials (ERPs) are promising for assisting individuals with motor impairments. However, their performance often degrades when transitioning from controlled laboratory settings to real-world environments. A significant factor in real-world environments is the presence of ambient sound, which can act as a distractor or cognitive load. While some studies have explored the effects of sound on auditory BCIs, the impact of sound on visual ERP-based BCIs, particularly how it influences mental workload and consequently performance, remains less understood. This paper aims to bridge this gap by systematically examining how varying auditory task difficulty (through speech speed) affects visual BCI performance.

Methods

The researchers designed a novel dual-task experiment where participants simultaneously engaged in a visual BCI task and an auditory task. The visual task involved identifying "target" robotic arm images within a matrix using an oddball paradigm, which elicits ERPs like P300 and N200. The auditory task consisted of listening to an audio story at three different speeds: no sound (0-T, low workload), half speed (0.5-T, medium workload), and normal speed (1-T, high workload). Participants were instructed to silently count specific words ("De") in the story while performing the visual BCI task.

Mental workload was assessed using the NASA-TLX scale. EEG data was collected and pre-processed, including filtering and baseline correction. ERPs were extracted and analyzed for amplitude and latency changes. Classification was performed using Fisher Linear Discriminant Analysis (FLDA). The study evaluated performance based on classification accuracy and Information Transfer Rate (ITR), and importantly, analyzed the impact of training and testing classifiers with data from the same or different workload conditions.

Datasets

The study utilized EEG data collected from 10 participants performing the described visual-auditory dual-task experiment. No pre-existing public datasets were used for training or evaluation. The experiment generated a novel dataset specific to the visual-auditory BCI paradigm under varying auditory workloads.

Key Insights and Ideas

  • Auditory Speed as a Workload Manipulator: The study effectively demonstrates that the speed of an auditory task can directly influence mental workload, impacting cognitive resource allocation.
  • Dual-Pathway Interaction: The research highlights the interconnectedness of auditory and visual pathways in the brain when performing a BCI task, suggesting that auditory input can interfere with visual processing.
  • ERP Amplitude as a Workload Indicator: Increased auditory workload leads to a significant decrease in P300 and N200 amplitudes, particularly in the occipital-parietal regions, serving as an objective measure of cognitive load.
  • Training-Testing Data Similarity is Crucial: Beyond ERP amplitude, the congruence between training and testing data (i.e., using data from similar workload conditions) is paramount for achieving high BCI performance. Mismatched data significantly degrades accuracy.

Key Results

  • ERP Amplitude Reduction: Increasing auditory task speed led to a decrease in P300 amplitude by 0.86 µV (p=0.0239) and N200 amplitude by 0.69 µV (p=0.0158) in the occipital-parietal area.
  • Performance Decline: This ERP amplitude reduction resulted in a 5.95% decline in accuracy (p=0.0101) and a 9.53 bits/min decline in Information Transfer Rate (ITR) (p=0.0416) as auditory speed increased.
  • Workload Impact: The study found that higher workload conditions (HD) generally led to lower accuracy and ITR compared to lower workload conditions (LD).
  • Training-Testing Match: Classifiers trained and tested on data from the same workload condition (e.g., LD-LD) consistently outperformed those trained and tested on different conditions (e.g., LD-MD, LD-HD), even if the ERP amplitudes were lower. For instance, LD-LD achieved 90.83% accuracy, while LD-HD achieved 76.39%.

Limitations and Future Work

The authors acknowledge that the study focused on a specific type of auditory stimulus (a story) and a particular visual BCI paradigm. Future work could explore the effects of different types of sounds (e.g., music, environmental noise) and other visual BCI paradigms. The study also noted that the sound of the robotic arm servos during online experiments might have had an additional effect, which warrants further investigation. Additionally, exploring how user familiarity with sounds or memory cognitive mechanisms might influence BCI performance is suggested for future research. The authors also propose investigating how to incorporate environmental noise into the training process to create more robust BCIs for real-world use.

Summary updated

Abstract

The performance of the event-related potential (ERP)-based brain-computer interface (BCI) declines when applying it into the real environment, which limits the generality of the BCI. The sound is a common noise in daily life, and whether it has influence on this decline is unknown. This study designs a visual-auditory BCI task that requires the subject to focus on the visual interface to output commands and simultaneously count number according to an auditory story. The story is played at three speeds to cause different workloads. Data collected under the same or different workloads are used to train and test classifiers. The results show that when the speed of playing the story increases, the amplitudes of P300 and N200 potentials decrease by 0.86 μV (<i>p</i> = 0.0239) and 0.69 μV (<i>p</i> = 0.0158) in occipital-parietal area, leading to a 5.95% decline (<i>p</i> = 0.0101) of accuracy and 9.53 bits/min decline (<i>p</i> = 0.0416) of information transfer rate. The classifier that is trained by the high workload data achieves higher accuracy than the one trained by the low workload if using the high workload data to test the performance. The result indicates that the sound could affect the visual ERP-BCI by increasing the workload. The large similarity of the training data and testing data is as important as the amplitudes of the ERP on obtaining high performance, which gives us an insight on how make to the ERP-BCI generalized.
BibTeX
@inproceedings{xu2020study,
  author = {Guizhi Xu and Yuwei Wu and Mengfan Li},
  title = {{The Study of Influence of Sound on Visual ERP-Based Brain Computer Interface}},
  booktitle = {Sensors},
  year = {2020},
  doi = {10.3390/s20041203},
}
Figure