Linear stimulus reconstruction works on the KU Leuven audiovisual, gaze-controlled auditory attention decoding dataset

Simon Geirnaert, Iustina Rotaru, Tom Francart, Alexander BertrandarXiv 2024 · DOI · arXiv

Abstract

Auditory Attention Decoding (AAD) aims to identify which speaker a listener is attending to using neural signals such as electroencephalography (EEG). Recent "spatial" AAD algorithms (Sp-AAD) have demonstrated high performance but were found to fail when eye gaze and auditory attention are decoupled. This study validates the KU Leuven audiovisual, gaze-controlled AAD (AV-GC-AAD) dataset using traditional linear stimulus reconstruction. We demonstrate that neural tracking of speech remains robust even under distracting visual conditions where gaze is incongruent with the auditory target. Our results establish a robust baseline for future AAD research and confirm that the AV-GC-AAD dataset is a reliable tool for evaluating algorithms without the interference of eye-gaze shortcuts.

Introduction

In complex acoustic environments, often referred to as the "cocktail party" scenario, humans possess the remarkable ability to focus on a single speaker while suppressing background noise and competing voices. Developing "smart" hearing aids that can replicate this ability requires robust Auditory Attention Decoding (AAD) algorithms. Recently, a new class of spatial AAD algorithms (Sp-AAD) claimed high accuracy by identifying the spatial location of the listener's attention. However, the AV-GC-AAD dataset revealed a critical flaw: these algorithms often rely on eye-gaze proxies rather than actual auditory processing. When participants look in one direction but listen in another, these models fail. This paper addresses whether this failure stems from dataset quality or algorithm design by applying a well-established linear stimulus reconstruction method to the AV-GC-AAD data.

Methodology: Linear Stimulus Reconstruction

The authors employ backward modeling, specifically linear stimulus reconstruction, to decode attention. This method attempts to reconstruct the "speech envelope"—the temporal volume profile of the audio—directly from the user's EEG signals.

Diagram of the linear stimulus reconstruction pipeline, showing how EEG signals are decoded to match the attended speaker's envelope.

As illustrated in the figure above, the pipeline consists of several stages:

  1. Envelope Extraction: The physical envelopes of the competing speech signals are extracted.
  2. EEG Decoding: A linear spatio-temporal filter (the decoder) is applied to 64-channel EEG data with time lags ranging from 0 to 400ms.
  3. Correlation Analysis: The reconstructed envelope is correlated with the envelopes of both the attended and unattended speakers.
  4. Classification: The speaker corresponding to the higher correlation coefficient ($\rho$) is identified as the attended speaker.

To prevent overfitting and ensure statistical validity, the study utilizes Ledoit-Wolf shrinkage for regularization and employs rigorous cross-validation techniques, including Leave-One-Trial-Out (LOTO) and Leave-One-Subject-Out (LOSO).

Experimental Setup and Datasets

The primary focus of this validation is the KU Leuven AV-GC-AAD dataset. This dataset includes EEG recordings from 16 subjects across four distinct experimental conditions designed to disentangle eye gaze from auditory attention.

Table of experimental conditions in the AV-GC-AAD dataset, detailing the relationship between auditory and visual attention.

As shown in the table above, the conditions vary in their degree of "congruence" between visual and auditory targets. By including "incongruent" tasks—such as following a moving crosshair while listening to a stationary speaker—the dataset ensures that decoding success must rely on auditory neural tracking rather than ocular artifacts. The study also tests the generalization of these decoders by training on the older KU Leuven AAD 2016 dataset and testing on the 2024 AV-GC-AAD data.

Key Results and Discussion

The application of linear stimulus reconstruction yielded several significant findings:

  • High Decoding Accuracy: Subject-specific decoding reached approximately 85-86% accuracy using 60-second decision windows.
  • Robustness Across Conditions: Significant, above-chance performance was achieved in every condition, including the most visually distracting ones. This proves that participants were indeed attending to the correct speaker as instructed, confirming the dataset's integrity.
  • Subject-Independent Success: The system successfully decoded attention for new subjects (LOSO-CV) with high significance, peaking at over 80% accuracy for 120-second windows.
  • The Role of Listening Effort: Interestingly, the "Moving target + noise" condition resulted in higher accuracies than the "Moving video" condition. This suggests that the presence of background babble noise may increase listening effort, leading to a more robust neural representation of the attended speech.

Conclusion

This study provides a definitive validation of the AV-GC-AAD dataset. By demonstrating that linear stimulus reconstruction—a method immune to eye-gaze shortcuts—works effectively, the authors prove that the dataset accurately captures neural auditory attention. This work establishes a "minimal benchmark" for the field, and the authors have released their MATLAB code to facilitate future comparisons. Future research, particularly involving Deep Learning models, must demonstrate an ability to outperform these linear baselines on this dataset without falling back on the "crutch" of eye-tracking. The next major challenge remains reducing the decision window to under 5 seconds to enable practical, real-time hearing aid applications.

Summary updated

Abstract

In a recent paper, we presented the KU Leuven audiovisual, gaze-controlled auditory attention decoding (AV-GC-AAD) dataset, in which we recorded electroencephalography (EEG) signals of participants attending to one out of two competing speakers under various audiovisual conditions. The main goal of this dataset was to disentangle the direction of gaze from the direction of auditory attention, in order to reveal gaze-related shortcuts in existing spatial AAD algorithms that aim to decode the (direction of) auditory attention directly from the EEG. Various methods based on spatial AAD do not achieve significant above-chance performances on our AV-GC-AAD dataset, indicating that previously reported results were mainly driven by eye gaze confounds in existing datasets. Still, these adverse outcomes are often discarded for reasons that are attributed to the limitations of the AV-GC-AAD dataset, such as the limited amount of data to train a working model, too much data heterogeneity due to different audiovisual conditions, or participants allegedly being unable to focus their auditory attention under the complex instructions. In this paper, we present the results of the linear stimulus reconstruction AAD algorithm and show that high AAD accuracy can be obtained within each individual condition and that the model generalizes across conditions, across new subjects, and even across datasets. Therefore, we eliminate any doubts that the inadequacy of the AV-GC-AAD dataset is the primary reason for the (spatial) AAD algorithms failing to achieve above-chance performance when compared to other datasets. Furthermore, this report provides a simple baseline evaluation procedure (including source code) that can serve as the minimal benchmark for all future AAD algorithms evaluated on this dataset.
BibTeX
@misc{geirnaert2024linear,
  author = {Simon Geirnaert and Iustina Rotaru and Tom Francart and Alexander Bertrand},
  title = {{Linear stimulus reconstruction works on the KU Leuven audiovisual, gaze-controlled auditory attention decoding dataset}},
  year = {2024},
  eprint = {2412.01401},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.01401},
}
Figure