Which Ones Are Speaking? Speaker-Inferred Model for Multi-Talker Speech Separation

Jing Shi, Jiaming Xu, Bo XuInterspeech 2019 · DOI

Introduction

The "Cocktail Party Problem"—the ability to isolate a single voice from a mixture of overlapping speakers—is a fundamental challenge in speech processing. While deep learning has significantly advanced multi-talker speech separation, two persistent issues remain: the permutation problem (correctly assigning output channels to speakers) and the output dimension mismatch problem (handling a variable number of speakers). Most existing models are constrained to a fixed number of outputs, often requiring prior knowledge of the speaker count.

This paper proposes a "speaker-inferred" model that leverages speaker identity as a prior. By first identifying who is speaking, the model can naturally determine how many are speaking and use these identities as anchors for separation.

Proposed Framework

The authors introduce a unified end-to-end framework composed of three distinct modules designed to handle inference, separation, and identity verification.

Overview of the speaker-inferred model architecture, highlighting the Inference, Separation, and Re-Identification modules.

1. Automatic Inference Module

Utilizing a Sequence-to-Sequence (Seq2Seq) architecture, this module processes the mixture spectrogram through a bidirectional LSTM encoder. An attention-based LSTM decoder then generates the identities of the speakers present. Because the decoder can output an <EOS> (End of Sequence) token, the model dynamically adapts to the number of speakers in the mixture, effectively counting them during the inference process.

2. Separation Module

The hidden embeddings (speaker "fingerprints") produced by the inference module serve as queries. An attention mechanism compares these fingerprints against the mixture's features to generate an Ideal Ratio Mask (IRM) for each identified speaker, effectively isolating their individual speech streams.

3. Re-Identification Module

During training, an optional re-identification module attempts to recognize the speaker from the separated audio. This serves as a regularization term, ensuring that the separation process preserves the unique acoustic characteristics of each speaker. Additionally, Schmidt orthogonalization is applied to speaker embeddings to ensure they remain distinct, preventing "bleeding" between output channels.

Experimental Results

The model was evaluated on the WSJ0-2mix dataset under both Closed Condition (CC) and Open Condition (OC) settings.

Comparison of the proposed model against state-of-the-art baselines in terms of SDR improvement and parameter efficiency.

Performance and Efficiency

As shown in the results table, the "speaker-inferred" model achieves highly competitive performance:

  • SDR Gains: On the WSJ0-2mix (CC), the model reached a Signal-to-Distortion Ratio (SDR) improvement of 11.0 dB, surpassing standard PIT-BLSTM (10.0 dB) and the previous TDAA baseline (9.1 dB).
  • Speaker Counting Accuracy: The inference module reached a 100% success rate in predicting the correct number of speakers in the 2-speaker test set, demonstrating robustness in speaker counting.
  • Model Compactness: With approximately 18.4M parameters, the model is significantly more efficient than high-performing uPIT-BLSTM variants, which can exceed 90M parameters.

Discussion and Future Directions

A key insight from this work is that identity acts as a powerful prior for signal separation. Visualization of the attention weights reveals that the model focuses on specific temporal frames where a particular speaker is most active, providing an interpretable link between identity and acoustic activity.

However, challenges remain. The current inference module performs classification over a known set of speakers, which may limit scalability in "in-the-wild" scenarios with thousands of unknown identities. Future work may explore more complex phase-sensitive masks and evaluate the model's performance in denser auditory environments with more than three overlapping speakers.

Summary updated

BibTeX
@inproceedings{shi2019which,
  author = {Jing Shi and Jiaming Xu and Bo Xu},
  title = {{Which Ones Are Speaking? Speaker-Inferred Model for Multi-Talker Speech Separation}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-1591},
}
Figure