Listen, Think and Listen Again: Capturing Top-down Auditory Attention for Speaker-independent Speech Separation

Jing Shi, Jiaming Xu, Guangcan Liu, Bo Xu

Abstract

Traditional speech separation models often struggle with the "permutation problem" and fixed output dimensions when dealing with an unknown number of speakers. This paper introduces the Top-Down Auditory Attention (TDAA) framework, which mimics the human ability to selectively attend to a single voice in a crowded environment. By integrating bottom-up inference for speaker identification with top-down attention for signal extraction, TDAA iteratively separates speakers one by one. Experimental results on the WSJ0 dataset show significant improvements in Signal-to-Distortion Ratio (SDR), demonstrating the model's robustness in speaker-independent scenarios.

Introduction

The "cocktail party effect"—the human ability to focus on a single speaker amidst background noise and competing voices—remains a significant challenge for automated speech separation systems. Most existing deep learning models attempt "direct separation," trying to isolate every voice in a mixture simultaneously. This approach faces two major hurdles: the permutation problem (label ambiguity during training) and the inability to handle a varying number of speakers.

Researchers from the Chinese Academy of Sciences (CASIA) propose a shift from these "driftless" strategies toward a human-centric "Listen, Think, and Listen Again" process. Their framework, Top-Down Auditory Attention (TDAA), treats speech separation as an iterative, selective attention task rather than a global optimization problem.

The TDAA Framework

The TDAA model consists of two primary components that work in a loop to extract individual speech signals.

Bottom-up Inference ("Thinking")

The process begins with a bottom-up analysis of the mixed speech spectrogram. A Bidirectional Long Short-Term Memory (BiLSTM) network acts as a classifier to identify "candidate speakers" within the mixture. Instead of relying on fixed identities, the model uses a set of "seeds"—learned voiceprints from the training set—to approximate the characteristics of the speakers present. This allows the model to generalize to unknown speakers by finding the closest matches in its learned embedding space.

Top-down Attention ("Listening Again")

Once a candidate speaker is identified, their specific embedding serves as a "query" for the attention mechanism. The model performs a dot-product attention between this query and the high-level features of the mixture to generate an Ideal Ratio Mask (IRM). This mask is then applied to the original mixture to extract the target speaker's voice.

The TDAA architecture integrates bottom-up inference for speaker identification and top-down attention for signal extraction.

Iterative Extraction and Recurrent Inference

To handle mixtures with an unknown number of speakers, TDAA employs a recurrent "extract-and-subtract" strategy. After the most salient speaker is extracted, their signal is subtracted from the mixture. The remaining signal is then fed back into the model to identify the next speaker.

The recurrent inference process iteratively extracts speakers and updates the remaining mixture until no more speakers are detected.

This iterative process continues until the classifier predicts "None," indicating that all significant speech components have been isolated. To prevent the model from repeatedly extracting the same speaker, the authors implemented Summarization Normalization, a loss function that encourages the generation of diverse and mutually exclusive masks.

Experimental Evaluation

The TDAA model was evaluated using the Wall Street Journal (WSJ0) mixed-speech tasks, covering both two-speaker (WSJ0-2mix) and three-speaker (WSJ0-3mix) scenarios.

Key Results

  • Speaker Independence: In "Open Condition" (OC) tests with completely unknown speakers, TDAA achieved a Signal-to-Distortion Ratio (SDR) improvement of 7.5 dB for two-speaker mixtures.
  • Multi-talker Flexibility: For three-speaker mixtures, the model achieved an SDR improvement of 4.6 dB, outperforming complex baselines like Deep Clustering (DC) that struggle with higher speaker counts.
  • Computational Efficiency: By focusing on one speaker at a time, TDAA avoids the factorial computational complexity ($N!$) associated with Permutation Invariant Training (PIT) as the number of speakers increases.

Conclusion

TDAA represents a significant step toward more biologically plausible and flexible speech separation. By mimicking the human top-down attention mechanism, the model solves the permutation problem and naturally adapts to varying numbers of speakers. While a performance gap remains between known and unknown speakers, the framework's ability to generalize using voiceprint "seeds" provides a robust foundation for future research in real-world, noisy environments.

Summary updated

Abstract

Recent deep learning methods have made significant progress in multi-talker mixed speech separation. However, most existing models adopt a driftless strategy to separate all the speech channels rather than selectively attend the target one. As a result, those frameworks may be failed to offer a satisfactory solution in complex auditory scene where the number of input sounds is usually uncertain and even dynamic. In this paper, we present a novel neural network based structure motivated by the top-down attention behavior of human when facing complicated acoustical scene. Different from previous works, our method constructs an inference-attention structure to predict interested candidates and extract each speech channel of them. Our work gets rid of the limitation that the number of channels must be given or the high computation complexity for label permutation problem. We evaluated our model on the WSJ0 mixed-speech tasks. In all the experiments, our model gets highly competitive to reach and even outperform the baselines.
BibTeX
@inproceedings{shi2018listen,
  author = {Jing Shi and Jiaming Xu and Guangcan Liu and Bo Xu},
  title = {{Listen, Think and Listen Again: Capturing Top-down Auditory Attention for Speaker-independent Speech Separation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.24963/ijcai.2018/605},
}
Figure