EEND-SS: Joint End-to-End Neural Speaker Diarization and Speech Separation for Flexible Number of Speakers

Soumi Maiti, Yushi Ueda, Shinji Watanabe, Chunlei Zhang, Meng Yu, Shi-Xiong Zhang, Yong Xu

Abstract

In complex acoustic environments involving multiple overlapping speakers, machines must simultaneously determine "who spoke when" (speaker diarization) and isolate individual voices (speech separation). Traditionally, these tasks are handled by separate, modular systems, often limited to a fixed number of speakers. This paper introduces EEND-SS, a unified end-to-end framework that jointly performs speaker diarization, counting, and speech separation for a variable number of speakers. By leveraging shared feature representations and a novel diarization-separation fusion mechanism, EEND-SS achieves state-of-the-art performance on both tasks while maintaining robustness in realistic, sparsely overlapped scenarios.

1. Introduction

The "cocktail party problem" remains a fundamental challenge in speech processing. While significant progress has been made in speaker diarization and speech separation independently, their inherent codependency is often overlooked in modular pipelines. Accurate diarization provides temporal cues that can guide separation, while clean, separated audio signals facilitate more precise speaker identification.

Furthermore, real-world conversations are dynamic; the number of active participants fluctuates, and speech is often "sparse," meaning speakers do not overlap continuously. Most existing separation models are constrained to a fixed number of outputs (e.g., two-speaker separation), limiting their utility in unpredictable environments like meetings or social gatherings. EEND-SS addresses these challenges by integrating diarization and separation into a single neural architecture capable of handling a flexible number of speakers.

2. Methodology

EEND-SS combines the strengths of Conv-TasNet for time-domain speech separation and EEND-EDA (End-to-End Neural Diarization with Encoder-Decoder Attractors) for diarization and speaker counting.

2.1 Shared Feature Backbone

The framework utilizes a shared encoder and a series of Temporal Convolutional Network (TCN) blocks. These blocks extract high-level "bottleneck features" that serve as the input for both the separation and diarization branches. This joint feature extraction allows the model to learn representations that are mutually beneficial for both tasks.

2.2 Flexible Speaker Handling

To accommodate a variable number of speakers without the computational overhead of multiple decoders, EEND-SS employs a dynamic selection strategy. The separation branch includes a bank of 1x1 convolutional layers. During inference, the diarization branch first estimates the number of speakers (k). The model then selects the corresponding k-th convolutional layer to generate the appropriate number of masks. This approach allows the heavy decoder weights to be shared across different speaker counts, improving parameter efficiency and generalization.

2.3 Diarization-Separation Fusion

A key innovation in EEND-SS is the fusion of diarization outputs with the separation process. The predicted speaker activity probabilities from the diarization branch are used to "gate" or mask the separated audio signals. This effectively acts as a learned Voice Activity Detection (VAD) filter, zeroing out residual noise or "bleeding" from other speakers during periods of silence.

2.4 Multitask Learning

The entire system is trained using a weighted sum of three losses: SI-SDR (separation), Diarization Error, and Attractor Existence (counting). This joint optimization ensures that the feature backbone learns to represent information relevant to all three objectives simultaneously.

3. Experimental Evaluation

The authors evaluated EEND-SS using standard benchmarks and a specialized "sparse" dataset to simulate real-world conditions.

3.1 Datasets

  • Libri2Mix & Libri3Mix: Standard benchmarks derived from LibriSpeech, containing mixtures of two and three speakers, respectively.
  • SparseLibriMix: A more realistic dataset where speakers do not overlap 100% of the time. It includes varying overlap ratios (0% to 100%) to test how well the model handles natural conversational flow.

3.2 Performance Metrics

The model demonstrated significant improvements across all primary metrics:

  • Separation Quality: On Libri2Mix, EEND-SS achieved an SI-SDRi of 11.20 dB, outperforming the standard Conv-TasNet baseline (10.82 dB).
  • Diarization Accuracy: The Diarization Error Rate (DER) dropped from 5.93% (baseline EEND-EDA) to 5.12% with joint training.
  • Speaker Counting: In mixtures containing a mix of 2 and 3 speakers, EEND-SS achieved a Speaker Counting Accuracy (SCA) of 98.2%, significantly higher than the EEND-EDA baseline (86.2%).

4. Discussion and Key Insights

The results confirm that joint training acts as a "force multiplier." The diarization branch benefits from the separation-oriented features, and vice versa. Notably, the model's performance on SparseLibriMix indicates that EEND-SS is particularly effective at handling realistic audio where speakers are not constantly talking over one another. The "selection" strategy for 1x1 conv layers proved to be an efficient way to handle variable speaker counts without sacrificing performance or increasing model size significantly.

5. Conclusion and Future Work

EEND-SS represents a significant step toward a truly unified solution for the cocktail party problem. By treating diarization and separation as a single, integrated task, the model achieves higher accuracy and greater flexibility than modular approaches.

Future research directions include incorporating massive self-supervised models like HuBERT or wav2vec 2.0 to push performance further. Additionally, while the current framework uses Conv-TasNet, the authors suggest the architecture is generalizable and could potentially incorporate newer backbones like DPRNN or Transformers. Finally, stress-testing the model on highly reverberant or extremely noisy real-world meeting data remains a priority for future development.

Summary updated

BibTeX
@inproceedings{maiti2022eend,
  author = {Soumi Maiti and Yushi Ueda and Shinji Watanabe and Chunlei Zhang and Meng Yu and Shi-Xiong Zhang and Yong Xu},
  title = {{EEND-SS: Joint End-to-End Neural Speaker Diarization and Speech Separation for Flexible Number of Speakers}},
  booktitle = {SLT},
  year = {2022},
  url = {https://openreview.net/forum?id=N91IDsnHqS1},
}
Figure