AADNet: An End-to-End Deep Learning Model for Auditory Attention Decoding

Nhan Duc Thanh Nguyen, Huy Phan, Simon Geirnaert, Kaare B. Mikkelsen, Preben Kidmose

TL;DR

This paper introduces AADNet, a novel end-to-end deep learning model for Auditory Attention Decoding (AAD) that directly predicts the attended speech from EEG signals, bypassing traditional multi-step approaches. AADNet significantly improves both subject-specific and subject-independent AAD performance, particularly in generalizing to unseen individuals, paving the way for more effective neuro-steered hearing devices.

What Problem Does This Paper Address?

Imagine being in a noisy room with multiple conversations happening simultaneously. Your brain has an amazing ability to focus on the one conversation you're interested in, filtering out the rest. This is called auditory attention. For individuals with hearing impairments, this ability is often compromised, making it difficult to follow conversations in such environments. Auditory Attention Decoding (AAD) aims to identify which speech stream a person is attending to by analyzing their brain signals, typically Electroencephalography (EEG). This information could be used to develop "neuro-steered" hearing aids that selectively amplify the attended speech and suppress background noise.

Current AAD methods often rely on a two-step process: first, predicting the "envelope" (a representation of the speech's loudness fluctuations) of the attended speech from EEG, and then correlating this prediction with the actual speech envelopes to identify the target. While effective, these methods can be complex and may not generalize well to new users. This paper addresses the need for a more direct, efficient, and generalizable AAD approach, especially for real-world applications like hearing assistance.

Methods

The core innovation of this paper is AADNet, an end-to-end deep learning model. Unlike previous methods that separate prediction and identification, AADNet directly maps EEG signals to the attended speech. It utilizes a modified Inception block architecture, inspired by image recognition models, which allows for parallel processing of features at different temporal scales.

AADNet has two main branches: one for EEG signals and one for audio envelopes. Both branches process their respective inputs through a series of convolutional layers within the Inception blocks, followed by pooling and batch normalization. Crucially, the model then computes the channel-wise Pearson correlation between the outputs of the EEG and audio branches. This correlation feature vector is then fed into fully connected layers and a softmax activation function to directly classify the attended speech stream. This direct approach aims to capture the complex relationship between brain activity and attended speech more effectively.

The paper also implements and compares AADNet against established baseline methods:

  • Linear Stimulus Reconstruction (LSR): A linear decoder that reconstructs the speech envelope from EEG.
  • Canonical Correlation Analysis (CCA): A method that finds optimal linear transformations for both EEG and audio to maximize their correlation, often followed by a classifier.
  • Non-linear Stimulus Reconstruction (NSR): A convolutional neural network (CNN) based approach that reconstructs the speech envelope non-linearly.

Datasets

The study utilizes three publicly available datasets, all recorded in competing-talker scenarios:

  • EventAAD: Contains data from 24 participants with 40 trials each, featuring Danish audiobooks/radio excerpts presented via loudspeakers.
  • DTU: Includes data from 18 healthy subjects with 60 trials each, using 50-second long speech segments in various acoustic conditions (anechoic, mild, and high reverberation) presented via insert earphones.
  • KUL: Features data from 16 subjects with 8+4 trials each, using 6-minute or 2-minute long Dutch stories presented via insert earphones.

The authors emphasize the importance of rigorous cross-validation, particularly a cross-trial leave-one-subject-out (LOSO) cross-validation scheme for subject-independent (SI) models, to prevent data leakage and ensure robust generalization.

Key Insights and Ideas

  • End-to-End Simplicity: Combining the prediction and identification stages into a single deep learning model simplifies the AAD pipeline and potentially improves performance by allowing the model to learn optimal feature representations directly.
  • Inception for Temporal Features: The use of Inception blocks with varying kernel sizes is effective in capturing diverse temporal patterns in EEG and audio signals, crucial for understanding speech encoding.
  • Pearson Correlation as a Bridge: Using Pearson correlation between the EEG and audio branch outputs provides a robust way to measure their relationship, forming a key feature for the final classification.
  • Subject-Independent Performance is Key: The paper strongly emphasizes the importance of subject-independent models for real-world applications, demonstrating that AADNet excels in generalizing to unseen individuals.
  • MESD as a Practical Metric: Beyond accuracy, the Minimal Expected Switch Duration (MESD) metric highlights AADNet's potential for real-time applications by showing it requires less time to adapt to attention shifts.

Key Results

  • Significant Accuracy Improvement: AADNet consistently outperformed baseline methods in both subject-specific (SS) and subject-independent (SI) settings across all three datasets.
  • Superior Generalization (SI Models): For SI models, AADNet achieved the highest accuracies, with average accuracies ranging from 56.1% to 82.7% across different window lengths and datasets. For instance, on the EventAAD dataset, AADNet reached 78.1% accuracy at a 20-second window length.
  • Reduced MESD: AADNet achieved significantly lower MESD values compared to other methods, indicating faster adaptation to attention shifts. For example, on the DTU dataset, AADNet's MESD was 12.3 seconds, compared to 18.1 seconds for NSR.
  • Window Length Matters: Performance generally increased with longer analysis window lengths, as expected, due to more available temporal information.

Limitations and Future Work

  • Short Window Length Performance: While AADNet excels at longer window lengths, its performance at very short durations (e.g., for MESD) was not superior to linear methods, suggesting room for improvement.
  • Data Requirements: Like most deep learning models, AADNet's performance is data-dependent, and limited data can lead to overfitting, especially for SS models. The authors addressed this by fine-tuning SI-trained models for SS tasks.
  • Comparison Scope: The study focused on specific types of AAD methods (envelope-based, SpkI task) and acknowledged that a comprehensive comparison with all existing AAD literature is challenging due to variations in datasets and methodologies.
  • Future Work: The authors suggest developing more advanced models to improve performance at short window lengths and exploring the potential of AADNet for real-time gain control systems in hearing-assistive devices. Further research into more sophisticated architectures and training strategies to mitigate data limitations is also encouraged.

Summary updated

Abstract

Auditory attention decoding (AAD) is the process of identifying the attended speech in a multi-talker environment using brain signals, typically recorded through electroencephalography (EEG). Over the past decade, AAD has undergone continuous development, driven by its promising application in neuro-steered hearing devices. Most AAD algorithms are relying on the increase in neural entrainment to the envelope of attended speech, as compared to unattended speech, typically using a two-step approach. First, the algorithm predicts representations of the attended speech signal envelopes; second, it identifies the attended speech by finding the highest correlation between the predictions and the representations of the actual speech signals. In this study, we proposed a novel end-to-end neural network architecture, named AADNet, which combines these two stages into a direct approach to address the AAD problem. We compare the proposed network against the traditional approaches, including linear stimulus reconstruction, canonical correlation analysis, and an alternative non-linear stimulus reconstruction using two different datasets. AADNet shows a significant performance improvement for both subject-specific and subject-independent models. Notably, the average subject-independent classification accuracies from 56.1 % to 82.7 % with analysis window lengths ranging from 1 to 40 seconds, respectively, show a significantly improved ability to generalize to data from unseen subjects. These results highlight the potential of deep learning models for advancing AAD, with promising implications for future hearing aids, assistive devices, and clinical assessments.
BibTeX
@misc{nguyen2024aadnet,
  author = {Nhan Duc Thanh Nguyen and Huy Phan and Simon Geirnaert and Kaare B. Mikkelsen and Preben Kidmose},
  title = {{AADNet: An End-to-End Deep Learning Model for Auditory Attention Decoding}},
  year = {2024},
  eprint = {2410.13059},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.13059},
}
Figure