Comparison of P300 Responses in Auditory, Visual and Audiovisual Spatial Speller BCI Paradigms

Moonjeong Chang, Nozomu Nishikawa, Zbigniew R. Struzik, Koichi Mori, Shoji Makino, Danilo P. Mandic, Tomasz M. RutkowskiarXiv 2013 · arXiv

Introduction

Brain-Computer Interfaces (BCIs) primarily utilize visual stimuli to detect the P300 event-related potential (ERP), a brain wave pattern associated with recognition and decision-making. While visual P300 spellers are highly effective, they can lead to user fatigue and are inaccessible to individuals with severe visual impairments. This research explores the integration of spatial cues—positioning stimuli in specific locations around the user—across visual, auditory, and combined audiovisual modalities to determine if multi-sensory or non-visual approaches can improve BCI performance for inexperienced users.

Methods

The study involved 16 BCI-naïve participants (average age ~22) who performed a spelling task involving five Japanese hiragana characters (a, i, u, e, o). Stimuli were presented from five distinct spatial locations (-45°, -22.5°, 0°, 22.5°, and 45°).

  • Visual Modality: Characters flashed at corresponding screen locations.
  • Auditory Modality: Virtual 3D soundstage created using vector-based sound amplitude panning, making sounds appear to originate from the five directions.
  • Audiovisual Modality: A simultaneous presentation of both visual and auditory stimuli.

EEG signals were captured using 16 active electrodes. The BCI2000 platform and a Linear Discrimination Analysis (LDA) classifier were employed to translate these signals into character selections based on the 0-600 ms window following stimulus presentation.

Results and Brain Signal Analysis

Analysis of the EEG data revealed a "latency paradox." The auditory unimodal paradigm produced a distinct P300 response with longer latencies compared to the visual and audiovisual modes. Furthermore, topographical mapping indicated that auditory P300 responses were more spatially localized, whereas visual tasks showed broader distributions across the scalp.

Topographical plots and ERP responses for visual (a), auditory (b), and audiovisual (c) modalities.

Despite the clear brain signatures observed in the auditory trials, this did not translate to superior performance. The visual and audiovisual modalities significantly outperformed the purely auditory approach in terms of spelling accuracy.

Mean accuracy and standard deviation across the three experimental modalities.

As shown in the results, the visual modality achieved a mean accuracy of 95.0%, and the audiovisual modality followed closely at 91.3%. In contrast, the auditory-only mode struggled with a mean accuracy of 52.5% and exhibited high variability among subjects (standard deviation of 27.2%).

Discussion and Conclusion

The findings suggest that while the brain generates a distinct and localized signal for spatial audio, current classification methods like LDA are not yet optimized to translate these specific ERP profiles into high-accuracy BCI performance. The audiovisual results indicate that adding sound to a visual speller does not significantly distract the user, maintaining high stability and accuracy.

Future research will focus on developing sophisticated signal processing techniques tailored to the "longer" auditory P300 response and refining virtual sound panning to enhance the distinctness of spatial cues. Currently, the auditory spatial speller remains a challenge for BCI novices compared to the robust performance of visual-based systems.

Summary updated

BibTeX
@misc{chang2013comparison,
  author = {Moonjeong Chang and Nozomu Nishikawa and Zbigniew R. Struzik and Koichi Mori and Shoji Makino and Danilo P. Mandic and Tomasz M. Rutkowski},
  title = {{Comparison of P300 Responses in Auditory, Visual and Audiovisual Spatial Speller BCI Paradigms}},
  year = {2013},
  eprint = {1301.6360},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1301.6360},
}
Figure