Voice Separation with an Unknown Number of Multiple Speakers

Eliya Nachmani, Yossi Adi, Lior Wolf

Abstract

Single-channel source separation, often referred to as the "Cocktail Party Problem," remains a significant challenge in audio processing, particularly as the number of concurrent speakers increases. We present a novel mask-free neural network architecture that leverages a gated RNN structure (MULCAT), multi-scale loss, and speaker-identity loss. Our approach demonstrates superior performance over existing state-of-the-art models, maintaining robustness even when the number of speakers is unknown and scales up to five.

Introduction

The ability to isolate individual voices from a single microphone recording is a complex task. Traditional "masking" methods, which learn to filter mixed audio representations, often struggle as more speakers are added, leading to performance degradation. Furthermore, many existing models require prior knowledge of the speaker count. This paper introduces a "mask-free" approach that directly generates separated signals and handles an unknown number of speakers through an automated activity detection mechanism.

Methodology

The core of the proposed architecture is the MULCAT (Multiply and Concatenate) block. Unlike standard RNNs, MULCAT utilizes Dual-Path RNNs to process audio chunks both locally and globally. The architecture employs a gated mechanism where outputs from parallel LSTMs are multiplied element-wise and concatenated with the input.

The overall architecture of the proposed mask-free separation model, featuring MULCAT blocks and a multi-scale reconstruction process.

To ensure high-quality separation and identity consistency, the model is trained using a triple-threat loss function:

  1. uPIT (utterance-level Permutation Invariant Training): Handles the arbitrary ordering of output channels.
  2. Multi-scale Loss: Reconstructs audio and calculates loss after every pair of RNN blocks to encourage incremental signal improvement.
  3. Speaker Classification Loss (ID Loss): Utilizes a pre-trained VGG11 network to maintain consistent speaker identity across each channel, preventing "identity switching."

Experimental Setup

The model was evaluated on standard benchmarks including WSJ0-2mix and WSJ0-3mix, as well as new, more challenging datasets: WSJ0-4mix and WSJ0-5mix. Robustness was further tested using WHAM! and WHAMR! datasets, which incorporate background noise and reverberation. Music separation performance was assessed using the MusDB dataset.

Results

The proposed model achieves a new state-of-the-art in voice separation. On the WSJ0-2mix benchmark, it reached an SI-SNRi of 20.1 dB. Notably, the performance gap between our model and previous methods like DPRNN widens as the number of speakers increases, demonstrating graceful degradation compared to the sharp "crashes" seen in competing architectures.

Comparison of the proposed model against existing state-of-the-art methods across different numbers of speakers.

For an unknown number of speakers, the "top-down" selection method—starting with a 5-speaker model and descending based on activity detection—proved highly accurate, matching the performance of models with known speaker counts.

Conclusion

By moving away from traditional masking and incorporating identity-preserving loss functions, this research provides a scalable solution to the Cocktail Party Problem. Future work will focus on further reducing identity switching and integrating speaker count estimation directly into a unified architecture to improve computational efficiency.

Summary updated

BibTeX
@inproceedings{nachmani2020voice,
  author = {Eliya Nachmani and Yossi Adi and Lior Wolf},
  title = {{Voice Separation with an Unknown Number of Multiple Speakers}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
}
Figure