Influence of Auditory Cues on the Neuronal Response to Naturalistic Visual Stimuli in a Virtual Reality Setting

George Al Boustani, Lennart J. K. Weiß, Hongwei Li, Svea Marie Meyer, Lukas Hiendlmeier, Philipp Rinklin, Bjoern Menze, Werner Hemmert, Bernhard WolfrumFrontiers in Human Neuroscience 2022 · DOI

TL;DR

This study investigates how adding auditory cues to realistic virtual reality (VR) visual stimuli affects brain responses and the performance of brain-computer interfaces (BCIs). They found that while auditory cues can suppress early visual brain responses (VEP), they can enhance later event-related potentials (ERPs), leading to improved classification accuracy for detecting more subtle or "burning" events. This highlights the importance of considering multisensory integration for robust BCIs in real-world applications.

What Problem Does This Paper Address?

Brain-computer interfaces (BCIs) hold immense promise for various applications, but their performance often degrades in real-world, complex environments compared to controlled lab settings. This is partly because real-world scenarios are inherently multimodal, involving simultaneous visual, auditory, and sometimes tactile information. Current BCI research, especially using VR, has largely focused on unimodal visual stimuli, neglecting how integrating other senses might influence neuronal processing and BCI efficacy. This paper aims to bridge this gap by exploring how adding congruent auditory cues to naturalistic visual events in VR impacts electroencephalography (EEG) signals and the subsequent ability to classify these events. Specifically, they investigate whether multisensory integration can improve the detection of safety-critical events in an industrial setting.

Methods

The researchers employed a virtual reality environment to simulate an industrial warehouse scenario. Participants experienced a conveyor belt system with two types of safety-related events: an "exploding box" and a "burning box," interspersed with normal operation (control trials). EEG data was recorded from 24 channels. The study was divided into two experimental conditions: visual-only and audio-visual, where matching auditory cues were presented alongside the visual stimuli.

Key methodological aspects include:

  • VR Environment: A custom-built VR scene using Blender and Unity to create realistic visual stimuli.
  • EEG Recording: A 24-channel portable EEG system with standard electrode placement.
  • Stimulus Design: Two distinct visual events (explosion and burning) designed to mimic varying degrees of severity and visual complexity.
  • Multisensory Integration: The audio-visual condition incorporated synchronized sounds that matched the visual events, including background noise to simulate an industrial environment.
  • Signal Processing: A comprehensive pipeline involving bad channel rejection, notch filtering, bandpass filtering, epoching around event onsets, local baseline correction, and independent component analysis (ICA) for artifact removal.
  • Feature Extraction: Three primary methods were used: variance (VAR), power spectral density (PSD), and discrete wavelet transform (DWT). These were also combined into a feature fusion method (DVP).
  • Classification: A Support Vector Machine (SVM) classifier with a linear kernel was employed for offline classification to detect the safety-relevant events (explosion/burning) against the control condition. Subject-independent classification was performed using k-fold cross-validation.
  • Statistical Analysis: Welch's t-tests were used to compare neuronal responses and classification accuracies between visual-only and audio-visual conditions.

Datasets

The study utilized a newly created dataset collected from 16 participants (9 in visual-only, 7 in audio-visual experiments). This dataset comprises EEG recordings from a controlled VR environment simulating industrial safety events. No pre-existing public datasets were used for training or evaluation.

Key Insights and Ideas

  • Severity Matters: The amplitude of brain responses (specifically VEPs and ERPs) directly correlated with the perceived severity of the visual event, with the more dramatic "explosion" eliciting stronger responses than the "burning box."
  • Auditory Cues Modulate Early Visual Processing: Surprisingly, adding congruent auditory cues to the "exploding box" stimulus significantly suppressed the early visual evoked potential (P1 component). This suggests an inhibitory mechanism at play in early sensory integration.
  • Auditory Cues Enhance Later Event Processing: Conversely, for the "burning box" stimulus, auditory cues led to an enhancement of later event-related potential components (N2c). This indicates that auditory input can boost the processing of more subtle or complex visual events.
  • Multisensory Integration Improves Detection of Subtle Events: The most significant finding for BCI applications is that adding auditory cues improved the classification accuracy for detecting the "burning box" event, suggesting that multisensory integration makes it easier for algorithms to distinguish less obvious events.
  • VR as a Realistic BCI Testbed: The study demonstrates the utility of VR for creating ecologically valid scenarios to study multisensory integration and its impact on BCI performance, bridging the gap between lab-controlled and real-world conditions.

Key Results

  • Neuronal Responses:
    • Visual-Only: Both explosion and burning stimuli elicited a VEP followed by an ERP over the occipital-parietal lobe. Amplitude reflected event severity.
  • Audio-Visual (Explosion): P1 component was significantly suppressed. N2c was similar in amplitude but delayed. P3b was delayed and increased.
  • Audio-Visual (Burning): P1 component was suppressed (~1.3x). N2c and P3b components were enhanced (~1.25x and ~1.5x, respectively) and delayed.
  • Classification Accuracy:
    • Explosion: Accuracy slightly decreased by -1.64% in the audio-visual condition compared to visual-only, particularly for DWT and DVP methods.
  • Burning Box: Accuracy increased by 5.58% in the audio-visual condition compared to visual-only, especially for PSD and DVP methods.
  • The DVP (feature fusion) method generally achieved the highest accuracy across both conditions and stimuli.

Limitations and Future Work

  • Small Sample Size: The study involved 16 participants (9 in visual-only, 7 in audio-visual), which is a limitation for statistical power and generalizability.
  • Subject Variability: Significant variability in amplitude and latency across subjects was observed, impacting statistical significance for some findings.
  • Specific VR Scenario: The findings are specific to the industrial warehouse scenario and the chosen stimuli. Future work could explore other environments and event types.
  • Future Directions: The authors suggest that future VR BCI studies should actively incorporate multisensory integration. They also highlight the potential for BCIs trained on multimodal input to achieve enhanced performance in real-world settings, especially for detecting less severe but critical events. Further research could explore the underlying neural mechanisms of this audio-visual interaction and its impact on different BCI paradigms.

Summary updated

Abstract

Virtual reality environments offer great opportunities to study the performance of brain-computer interfaces (BCIs) in real-world contexts. As real-world stimuli are typically multimodal, their neuronal integration elicits complex response patterns. To investigate the effect of additional auditory cues on the processing of visual information, we used virtual reality to mimic safety-related events in an industrial environment while we concomitantly recorded electroencephalography (EEG) signals. We simulated a box traveling on a conveyor belt system where two types of stimuli - an exploding and a burning box - interrupt regular operation. The recordings from 16 subjects were divided into two subsets, a visual-only and an audio-visual experiment. In the visual-only experiment, the response patterns for both stimuli elicited a similar pattern - a visual evoked potential (VEP) followed by an event-related potential (ERP) over the occipital-parietal lobe. Moreover, we found the perceived severity of the event to be reflected in the signal amplitude. Interestingly, the additional auditory cues had a twofold effect on the previous findings: The P1 component was significantly suppressed in the case of the exploding box stimulus, whereas the N2c showed an enhancement for the burning box stimulus. This result highlights the impact of multisensory integration on the performance of realistic BCI applications. Indeed, we observed alterations in the offline classification accuracy for a detection task based on a mixed feature extraction (variance, power spectral density, and discrete wavelet transform) and a support vector machine classifier. In the case of the explosion, the accuracy slightly decreased by -1.64% p. in an audio-visual experiment compared to the visual-only. Contrarily, the classification accuracy for the burning box increased by 5.58% p. when additional auditory cues were present. Hence, we conclude, that especially in challenging detection tasks, it is favorable to consider the potential of multisensory integration when BCIs are supposed to operate under (multimodal) real-world conditions.
BibTeX
@inproceedings{boustani2022influence,
  author = {George Al Boustani and Lennart J. K. Weiß and Hongwei Li and Svea Marie Meyer and Lukas Hiendlmeier and Philipp Rinklin and Bjoern Menze and Werner Hemmert and Bernhard Wolfrum},
  title = {{Influence of Auditory Cues on the Neuronal Response to Naturalistic Visual Stimuli in a Virtual Reality Setting}},
  booktitle = {Frontiers in Human Neuroscience},
  year = {2022},
  doi = {10.3389/fnhum.2022.809293},
}
Figure