Audio-Visual Target Speaker Extraction With Selective Auditory Attention

Ruijie Tao, Xinyuan Qian, Yidi Jiang, Junjie Li, Jiadong Wang, Haizhou LiIEEE Transactions on Audio, Speech and Language Processing 2025 · OpenReview

Abstract

Audio-Visual Target Speaker Extraction (AV-TSE) aims to isolate a specific speaker's voice from a noisy mixture by leveraging synchronized visual cues, such as lip movements. Traditional AV-TSE frameworks primarily utilize an "extraction-only" strategy, which focuses on identifying audio segments that correlate with visual data but often fails to account for the specific characteristics of background interference. This paper introduces the Subtraction-and-ExtrAction network (SEANet), a novel framework that mimics the human brain's selective auditory attention by explicitly modeling both target speech and background noise. By employing a unique Reverse Attention Module, SEANet penalizes correlations between estimated speech and noise, ensuring mutual exclusivity. Experimental results demonstrate that SEANet achieves state-of-the-art performance on the LRS2 and VoxCeleb2 datasets with significantly higher parameter efficiency than existing Transformer-based baselines.

Introduction

In complex acoustic environments—such as a crowded cafe or a busy street—humans possess the remarkable ability to focus on a single conversation while tuning out background noise, a phenomenon known as the "cocktail party effect." In the field of machine learning, Audio-Visual Target Speaker Extraction (AV-TSE) attempts to replicate this ability by using a target speaker's lip movements to guide the extraction of their clean speech from a multi-talker or noisy audio stream.

Despite recent progress, most current AV-TSE models rely on a singular extraction strategy: they search the audio signal for patterns that synchronize with the visual input. However, this approach often ignores the interference itself. Without explicitly modeling what not to listen to, these models frequently struggle in high-noise environments or when faced with competing speakers, leading to the accidental extraction of artifacts or incorrect voices.

The SEANet Framework

To address these limitations, we propose the Subtraction-and-ExtrAction network (SEANet). Unlike traditional models, SEANet treats noise as an informative signal rather than an invisible byproduct. The architecture is built around the concept of Parallel Speech and Noise Learning (PSNL), which utilizes dual branches to estimate the target speech and the background interference simultaneously.

Reverse Selective Auditory Attention

The core innovation of SEANet is its Reverse Attention Module. While standard self-attention mechanisms are used in the speech branch to maintain temporal consistency of the voice, the Reverse Attention Module computes the cross-attention between the estimated speech and the estimated noise.

If a specific segment of the extracted speech shows a high correlation with the estimated noise, the model identifies this as a leakage of interference. The reverse attention mechanism then assigns a negative score to these segments, actively penalizing and suppressing them. This mathematical enforcement of "speech-noise exclusivity" ensures that the network learns a cleaner, more distinct representation of the target speaker.

Experimental Setup

SEANet was rigorously evaluated using several benchmark datasets to test both in-domain performance and cross-domain generalization:

  • Primary Training/Evaluation: LRS2 (BBC clips) and VoxCeleb2 (YouTube interviews) provided large-scale, real-world data.
  • Cross-Domain Testing: To ensure robustness, models trained on VoxCeleb2 were tested on unseen datasets, including LRS3 (TED talks), Grid (studio multi-talker), and TCD-TIMIT (high-quality studio video).
  • Noise Robustness: The MUSAN dataset was utilized to augment audio with non-speech environmental noises, such as music and ambient sounds.

Results and Discussion

State-of-the-Art Performance

SEANet demonstrated superior extraction quality across all major benchmarks. On the LRS2 dataset, the model achieved a Scale-Invariant Signal-to-Distortion Ratio (SI-SDR) of 13.08 dB and a Word Error Rate (WER) of 12.26%. This outperforms heavy-duty models like AV-Sepformer, which reached 12.60 dB SI-SDR and 12.82% WER.

Architectural Efficiency

One of the most significant advantages of SEANet is its efficiency. While many modern models rely on brute-force scaling of Transformers, SEANet achieves its results with only 8.70M parameters and 21.17G FLOPs. In comparison, MuSE requires 15.01M parameters, and AV-Sepformer requires 21.73M parameters. This suggests that incorporating the right inductive biases—specifically the subtraction of noise—is more effective than simply increasing model depth.

Robustness and Generalization

Ablation studies confirmed that the reverse attention mechanism is critical for reducing "incorrect extraction" errors. Furthermore, SEANet maintained its performance advantage during cross-domain testing on the LRS3 and Grid datasets, proving that the subtraction-and-extraction strategy generalizes well to different acoustic and visual environments.

Conclusion and Future Work

SEANet demonstrates that explicitly modeling background noise through a reverse attention mechanism significantly improves the precision of target speaker extraction. By treating noise as a reference to be subtracted rather than a signal to be ignored, the model achieves state-of-the-art results with high computational efficiency.

Future research will focus on adapting this selective auditory attention mechanism for audio-only scenarios where visual data is unavailable. Additionally, the team aims to explore the integration of other auxiliary cues, such as pre-enrolled speaker voiceprints, and further optimize early-stage audio-visual fusion techniques to handle the synchronization flaws often found in "in-the-wild" datasets.

Summary updated

Abstract

Audio-visual target speaker extraction (AV-TSE) aims to extract the specific person's speech from the audio mixture given auxiliary visual cues. Previous methods usually search for the target voice through speech-lip synchronization. However, this strategy mainly focuseson the existence of target speech, while ignoring the variations of the noise characteristics, i.e., interference speaker and the background noise. That may result in extracting noisy signals from the incorrect sound source in challenging acoustic situations. To this end, we propose a novel selective auditory attention mechanism, which can suppress interference speakers and non-speech signals to avoid incorrect speaker extraction. By estimating and utilizing the undesired noisy signal through this mechanism, we design an AV-TSE framework named Subtraction-and-ExtrAction network (SEANet) to suppress the noisy signals. We conduct abundant experiments by re-implementing three popular AV-TSE methods as the baselines and involving nine metrics for evaluation. The experimental results show that our proposed SEANet achieves state-of-the-art results and performs well for all five datasets.
BibTeX
@article{tao2025audio,
  author = {Ruijie Tao and Xinyuan Qian and Yidi Jiang and Junjie Li and Jiadong Wang and Haizhou Li},
  title = {{Audio-Visual Target Speaker Extraction With Selective Auditory Attention}},
  journal = {IEEE Transactions on Audio, Speech and Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=ZZK4lkMlob},
}
Figure