Introduction
The "Cocktail Party Problem"—the ability to isolate a single voice from a mixture of overlapping speakers—is a fundamental challenge in speech processing. While deep learning has significantly advanced multi-talker speech separation, two persistent issues remain: the permutation problem (correctly assigning output channels to speakers) and the output dimension mismatch problem (handling a variable number of speakers). Most existing models are constrained to a fixed number of outputs, often requiring prior knowledge of the speaker count.
This paper proposes a "speaker-inferred" model that leverages speaker identity as a prior. By first identifying who is speaking, the model can naturally determine how many are speaking and use these identities as anchors for separation.
Proposed Framework
The authors introduce a unified end-to-end framework composed of three distinct modules designed to handle inference, separation, and identity verification.
1. Automatic Inference Module
Utilizing a Sequence-to-Sequence (Seq2Seq) architecture, this module processes the mixture spectrogram through a bidirectional LSTM encoder. An attention-based LSTM decoder then generates the identities of the speakers present. Because the decoder can output an <EOS> (End of Sequence) token, the model dynamically adapts to the number of speakers in the mixture, effectively counting them during the inference process.
2. Separation Module
The hidden embeddings (speaker "fingerprints") produced by the inference module serve as queries. An attention mechanism compares these fingerprints against the mixture's features to generate an Ideal Ratio Mask (IRM) for each identified speaker, effectively isolating their individual speech streams.
3. Re-Identification Module
During training, an optional re-identification module attempts to recognize the speaker from the separated audio. This serves as a regularization term, ensuring that the separation process preserves the unique acoustic characteristics of each speaker. Additionally, Schmidt orthogonalization is applied to speaker embeddings to ensure they remain distinct, preventing "bleeding" between output channels.
Experimental Results
The model was evaluated on the WSJ0-2mix dataset under both Closed Condition (CC) and Open Condition (OC) settings.
Performance and Efficiency
As shown in the results table, the "speaker-inferred" model achieves highly competitive performance:
- SDR Gains: On the WSJ0-2mix (CC), the model reached a Signal-to-Distortion Ratio (SDR) improvement of 11.0 dB, surpassing standard PIT-BLSTM (10.0 dB) and the previous TDAA baseline (9.1 dB).
- Speaker Counting Accuracy: The inference module reached a 100% success rate in predicting the correct number of speakers in the 2-speaker test set, demonstrating robustness in speaker counting.
- Model Compactness: With approximately 18.4M parameters, the model is significantly more efficient than high-performing uPIT-BLSTM variants, which can exceed 90M parameters.
Discussion and Future Directions
A key insight from this work is that identity acts as a powerful prior for signal separation. Visualization of the attention weights reveals that the model focuses on specific temporal frames where a particular speaker is most active, providing an interpretable link between identity and acoustic activity.
However, challenges remain. The current inference module performs classification over a known set of speakers, which may limit scalability in "in-the-wild" scenarios with thousands of unknown identities. Future work may explore more complex phase-sensitive masks and evaluate the model's performance in denser auditory environments with more than three overlapping speakers.
Summary updated