DBPNet: Dual-Branch Parallel Network with Temporal-Frequency Fusion for Auditory Attention Detection

Qinke Ni, Hongyu Zhang, Cunhang Fan

Abstract

Auditory Attention Decoding (AAD) aims to identify the focus of attention from EEG signals, a task complicated by low signal-to-noise ratios and the need for low-latency processing. We propose DBPNet, a dual-branch parallel network that fuses temporal features via a Transformer-based Temporal Attentive Branch (TABNet) and spectral-spatial features via a Frequency Residual Branch (FRBNet). DBPNet achieves significant accuracy improvements on ultra-short 0.1s decision windows while maintaining a lightweight architecture suitable for edge devices like hearing aids.

Introduction

In crowded "cocktail party" environments, humans naturally filter background noise to focus on a single speaker. For individuals with hearing impairments, modern hearing aids struggle to replicate this selective attention. Auditory Attention Decoding (AAD) attempts to solve this by analyzing Electroencephalography (EEG) signals. However, EEG data is noisy and non-linear, and existing models often require long decision windows that create unacceptable latency for real-time use. DBPNet addresses these challenges by fusing temporal and frequency domain information in a highly efficient architecture.

Methodology

DBPNet utilizes a dual-branch architecture to capture the multi-faceted nature of neural responses to auditory stimuli:

  1. Temporal Attentive Branch (TABNet): Employs a Transformer encoder to weight specific moments in the EEG sequence that indicate attention, capturing the rhythmic brain responses to speech.
  2. Frequency Residual Branch (FRBNet): Focuses on spectral-spatial information by decomposing EEG into five frequency bands (delta, theta, alpha, beta, and gamma) and calculating Differential Entropy (DE). These are projected onto 2D topological maps and processed using 3D Residual Convolutional blocks.
The architecture of DBPNet, featuring the Temporal Attentive Branch (TABNet) and the Frequency Residual Branch (FRBNet).

Before processing, the Common Spatial Pattern (CSP) algorithm is used to enhance the signal-to-noise ratio. The features from both branches are concatenated for final classification.

Key Results

DBPNet was evaluated on the KUL, DTU, and a new multi-modal MM-AAD dataset. Key findings include:

  • Superior Accuracy at Low Latency: On the MM-AAD dataset, DBPNet achieved 91.4% accuracy with a 0.1-second decision window, a 20.4% relative improvement over state-of-the-art baselines.
  • Extreme Efficiency: The model uses only 0.91 million parameters—91 times fewer than the baseline MBSSFCC model—making it ideal for low-power edge devices.
  • Domain Fusion: Ablation studies confirmed that combining temporal dynamics with physiological brain states provides a more robust neural signature of attention than either domain alone.
Comparison of decoding accuracy across different decision window lengths for TABNet, FRBNet, and the full DBPNet model.

Conclusion

DBPNet demonstrates that fusing temporal and frequency features significantly boosts AAD performance, particularly at the low latencies required for real-time hearing aid applications. Future work will explore Graph Neural Networks (GNNs) to better model the brain's 3D non-Euclidean connectivity and test the system in more diverse, real-world acoustic environments.

Summary updated

BibTeX
@inproceedings{ni2024dbpnet,
  author = {Qinke Ni and Hongyu Zhang and Cunhang Fan},
  title = {{DBPNet: Dual-Branch Parallel Network with Temporal-Frequency Fusion for Auditory Attention Detection}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
}
Figure