Abstract
Traditional speech separation models often struggle with the "permutation problem" and fixed output dimensions when dealing with an unknown number of speakers. This paper introduces the Top-Down Auditory Attention (TDAA) framework, which mimics the human ability to selectively attend to a single voice in a crowded environment. By integrating bottom-up inference for speaker identification with top-down attention for signal extraction, TDAA iteratively separates speakers one by one. Experimental results on the WSJ0 dataset show significant improvements in Signal-to-Distortion Ratio (SDR), demonstrating the model's robustness in speaker-independent scenarios.
Introduction
The "cocktail party effect"—the human ability to focus on a single speaker amidst background noise and competing voices—remains a significant challenge for automated speech separation systems. Most existing deep learning models attempt "direct separation," trying to isolate every voice in a mixture simultaneously. This approach faces two major hurdles: the permutation problem (label ambiguity during training) and the inability to handle a varying number of speakers.
Researchers from the Chinese Academy of Sciences (CASIA) propose a shift from these "driftless" strategies toward a human-centric "Listen, Think, and Listen Again" process. Their framework, Top-Down Auditory Attention (TDAA), treats speech separation as an iterative, selective attention task rather than a global optimization problem.
The TDAA Framework
The TDAA model consists of two primary components that work in a loop to extract individual speech signals.
Bottom-up Inference ("Thinking")
The process begins with a bottom-up analysis of the mixed speech spectrogram. A Bidirectional Long Short-Term Memory (BiLSTM) network acts as a classifier to identify "candidate speakers" within the mixture. Instead of relying on fixed identities, the model uses a set of "seeds"—learned voiceprints from the training set—to approximate the characteristics of the speakers present. This allows the model to generalize to unknown speakers by finding the closest matches in its learned embedding space.
Top-down Attention ("Listening Again")
Once a candidate speaker is identified, their specific embedding serves as a "query" for the attention mechanism. The model performs a dot-product attention between this query and the high-level features of the mixture to generate an Ideal Ratio Mask (IRM). This mask is then applied to the original mixture to extract the target speaker's voice.
Iterative Extraction and Recurrent Inference
To handle mixtures with an unknown number of speakers, TDAA employs a recurrent "extract-and-subtract" strategy. After the most salient speaker is extracted, their signal is subtracted from the mixture. The remaining signal is then fed back into the model to identify the next speaker.
This iterative process continues until the classifier predicts "None," indicating that all significant speech components have been isolated. To prevent the model from repeatedly extracting the same speaker, the authors implemented Summarization Normalization, a loss function that encourages the generation of diverse and mutually exclusive masks.
Experimental Evaluation
The TDAA model was evaluated using the Wall Street Journal (WSJ0) mixed-speech tasks, covering both two-speaker (WSJ0-2mix) and three-speaker (WSJ0-3mix) scenarios.
Key Results
- Speaker Independence: In "Open Condition" (OC) tests with completely unknown speakers, TDAA achieved a Signal-to-Distortion Ratio (SDR) improvement of 7.5 dB for two-speaker mixtures.
- Multi-talker Flexibility: For three-speaker mixtures, the model achieved an SDR improvement of 4.6 dB, outperforming complex baselines like Deep Clustering (DC) that struggle with higher speaker counts.
- Computational Efficiency: By focusing on one speaker at a time, TDAA avoids the factorial computational complexity ($N!$) associated with Permutation Invariant Training (PIT) as the number of speakers increases.
Conclusion
TDAA represents a significant step toward more biologically plausible and flexible speech separation. By mimicking the human top-down attention mechanism, the model solves the permutation problem and naturally adapts to varying numbers of speakers. While a performance gap remains between known and unknown speakers, the framework's ability to generalize using voiceprint "seeds" provides a robust foundation for future research in real-world, noisy environments.
Summary updated