Classification of Targets and Distractors in an Audiovisual Attention Task Based on Electroencephalography

Steven Mortier, Renata Turkes, Jorg De Winne, Wannes Van Ransbeeck, Dick Botteldooren, Paul Devos, Steven Latré, Marc Leman, Tim Verdonck

TL;DR

This paper presents a machine learning approach to classify target vs. distractor stimuli from EEG data during an audiovisual attention task. They achieved up to 76% accuracy using various ML models, demonstrating that attention-related brain signals (specifically P3a ERPs) can be reliably detected and used for brain-computer interfaces.

What Problem Does This Paper Address?

The paper tackles the challenge of improving human-AI interaction by enhancing synchronization and engagement. Specifically, it investigates whether auditory and rhythmic cues can boost attention to visual stimuli in a busy information stream. The core problem addressed is the accurate detection of attended (target) versus unattended (distractor) visual stimuli using electroencephalography (EEG) data, with the ultimate goal of developing more effective brain-computer interfaces (BCIs). This is crucial because current BCIs often require extensive user-specific calibration, a process that is time-consuming and inconvenient.

Methods

The researchers employed a suite of machine learning (ML) models to classify EEG data. They explored both individual-subject (IS) models, trained and tested on data from the same participant, and cross-subject (CS) models, designed to generalize across different individuals. The ML approaches included:

  • xDAWN + RG: A feature-based method that uses spatial filtering (xDAWN) and Riemannian geometry for classification.
  • EEGNet: A deep learning model (specifically a convolutional neural network or CNN) that performs both feature extraction and classification in an end-to-end manner.
  • Rocket and MiniRocket: General time-series classification methods that use random convolutional kernels.

To understand how the models made their decisions, they utilized explainable AI (xAI) techniques, specifically saliency maps. These maps highlight which parts of the input data (EEG channels and time points) were most influential for the model's prediction.

Datasets

The study utilized a novel dataset collected during the WithMe experiment. This experiment involved 42 young adults performing a modified digit-span task with varying levels of auditory and rhythmic support. The dataset consists of EEG recordings from 64 electrodes, sampled at 2048 Hz, and preprocessed to remove artifacts and downsampled to 50 Hz. The data was epoched into 1.2-second segments. The paper notes that some ground truth labels might have been imperfect due to potential mismatches between experimental design and subject perception.

Key Insights and Ideas

  • Auditory support aids attention detection: The study suggests that auditory cues, particularly in conditions with rhythmic support, can enhance the detectability of target stimuli in EEG data. This is evidenced by slightly higher accuracies in conditions with auditory support.
  • P3a ERP is a reliable attention marker: The findings strongly support the hypothesis that the P3a event-related potential (ERP), a known neural signature of attention, is a key feature that ML models leverage for distinguishing targets from distractors.
  • Generalization is achievable, but challenging: While IS models generally performed better, CS models achieved competitive results, indicating that generalization across subjects is possible, especially with more complex models like EEGNet. This is a critical step towards practical, user-independent BCIs.
  • Explainable AI is crucial for understanding BCI models: Saliency maps provided valuable insights, confirming that models focus on expected brain regions (parietal-occipital) and time windows (200-300 ms post-stimulus) associated with the P3a ERP, thus validating the biological plausibility of the model's predictions.

Key Results

  • Classification Accuracy: The models achieved classification accuracies of up to 76% for both IS and CS scenarios.
  • Model Performance: EEG-specific models (xDAWN + RG and EEGNet) generally outperformed general time-series models (Rocket, MiniRocket) for IS classification. For CS classification, EEGNet showed a slight edge over xDAWN + RG.
  • Computational Efficiency: While EEGNet achieved strong performance, it was significantly more computationally expensive to train than xDAWN + RG. MiniRocket offered a much faster training time compared to Rocket.
  • Saliency Map Findings: Saliency maps consistently highlighted the parietal-occipital electrodes and time points between 200 ms and 300 ms after stimulus onset as crucial for target detection, aligning with expectations for P3a ERPs.
  • Performance Drop: The drop in performance from IS to CS models was less pronounced than anticipated, particularly for EEGNet, which was attributed to the larger training dataset available for CS models.

Limitations and Future Work

  • Labeling Imperfections: The paper acknowledges that some ground truth labels in the dataset might have been imperfect, potentially due to subjects not correctly remembering sequences or misidentifying stimuli. Future work could involve filtering data to include only perfectly recalled sequences or removing "bad sequences."
  • Experiment Design: For future work, an experiment specifically designed for attention detection, with labels directly reflecting the subject's perception, could lead to more accurate attention detectors.
  • Feature Selection and Transfer Learning: The authors suggest exploring exhaustive feature selection and transfer learning techniques to further improve CS generalization, potentially bringing lightweight models like xDAWN + RG closer to the performance of more complex models.
  • Working Memory Investigation: The collected data can also be used to study working memory, particularly how it is influenced by auditory and rhythmic support.

Summary updated

Abstract

Within the broader context of improving interactions between artificial intelligence and humans, the question has arisen regarding whether auditory and rhythmic support could increase attention for visual stimuli that do not stand out clearly from an information stream. To this end, we designed an experiment inspired by pip-and-pop but more appropriate for eliciting attention and P3a-event-related potentials (ERPs). In this study, the aim was to distinguish between targets and distractors based on the subject’s electroencephalography (EEG) data. We achieved this objective by employing different machine learning (ML) methods for both individual-subject (IS) and cross-subject (CS) models. Finally, we investigated which EEG channels and time points were used by the model to make its predictions using saliency maps. We were able to successfully perform the aforementioned classification task for both the IS and CS scenarios, reaching classification accuracies up to 76%. In accordance with the literature, the model primarily used the parietal–occipital electrodes between 200 ms and 300 ms after the stimulus to make its prediction. The findings from this research contribute to the development of more effective P300-based brain–computer interfaces. Furthermore, they validate the EEG data collected in our experiment.
BibTeX
@inproceedings{mortier2023classification,
  author = {Steven Mortier and Renata Turkes and Jorg De Winne and Wannes Van Ransbeeck and Dick Botteldooren and Paul Devos and Steven Latré and Marc Leman and Tim Verdonck},
  title = {{Classification of Targets and Distractors in an Audiovisual Attention Task Based on Electroencephalography}},
  booktitle = {Sensors},
  year = {2023},
  url = {https://openreview.net/forum?id=ySjaFq2e8E},
}
Figure