EEND-DEMUX: End-to-End Neural Speaker Diarization via Demultiplexed Speaker Embeddings

Sung Hwan Mun, Min Hyun Han, Chanyeong Moon, Nam Soo KimarXiv 2023 · DOI · arXiv

Abstract

EEND-DEMUX is a novel end-to-end neural speaker diarization model designed to improve "who spoke when" detection by disentangling speaker-specific information directly in the latent embedding space. By employing a "demultiplexing" operation alongside specific structural constraints—matching, orthogonality, and sparsity—the model achieves significantly lower error rates than previous state-of-the-art methods on both fixed and flexible numbers of speakers.

Introduction

Speaker diarization is the process of partitioning an audio stream into segments based on speaker identity. Traditional systems typically rely on clustering-based pipelines involving separate modules for voice activity detection, segmentation, and embedding extraction. These systems often fail during "overlapping speech"—when multiple individuals speak simultaneously—because they usually assign a time segment to only one person.

End-to-End Neural Diarization (EEND) was developed to address this by predicting multi-speaker activity directly. However, standard EEND models often struggle with a flexible number of speakers or require complex auxiliary tasks like full speech separation. EEND-DEMUX bridges this gap by disentangling speakers in the neural network's internal representation without the overhead of full audio separation.

Methods: The EEND-DEMUX Architecture

The EEND-DEMUX model consists of four main components designed to extract and separate speaker information efficiently:

  1. MixtureEncoder: A Transformer-based encoder that processes raw audio features into "mixture embeddings" containing information from all speakers.
  2. Demultiplexer: The core novelty of the system. It uses parallel 1D-CNN modules to split the mixture embedding into $S$ separate "demultiplexed" speaker embeddings at every frame.
  3. AttractorDecoder: This module uses Multi-Head Cross-Attention (MHCA) to find "attractors" (representative vectors for each speaker) by analyzing the correlation between the mixture and the separated embeddings.
  4. SpeakerEncoder: A pre-trained network that provides "oracle" speaker embeddings during training to guide the demultiplexing process.
Architecture of the EEND-DEMUX model, illustrating the flow from mixture embeddings to demultiplexed speaker outputs.

Structural Constraints

To optimize the demultiplexing process, the authors introduce a total loss function with three unique constraints:

  • Matching (Distillation): Forces the demultiplexed embeddings to align with the "clean" oracle embeddings.
  • Orthogonality: Ensures embeddings from different speakers are mathematically distinct while embeddings from the same speaker remain similar.
  • Sparsity: Uses an $\ell_1$-norm penalty to encourage the model to focus on the most important features, reducing overfitting.

Key Results

The EEND-DEMUX model was evaluated using the LibriMix dataset, a standard benchmark for overlapping speech. The model consistently outperformed baseline EEND models across all scenarios.

Experimental results on the Libri2Mix dataset, showing the impact of adding distillation, orthogonality, and sparsity losses.

As shown in the results for the 2-speaker scenario, the combination of structural constraints leads to a significant reduction in the Diarization Error Rate (DER). Key performance highlights include:

  • 2-Speaker Scenario: Achieved a DER of 3.79%, a relative improvement of 24.50% over the baseline.
  • 3-Speaker Scenario: Achieved a DER of 4.91%, a relative improvement of 18.17%.
  • Flexible Speakers: On the combined Libri2&3Mix dataset, the model achieved a DER of 4.39%, outperforming state-of-the-art models like TS-VAD and USED.

Conclusion

EEND-DEMUX demonstrates that disentangling speakers in the embedding space is a highly effective strategy for end-to-end diarization. By avoiding the computational cost of full signal reconstruction while maintaining the benefits of multi-task learning, the model sets a new standard for handling overlapping speech. Future work will explore integrating this framework into larger multi-talker systems, such as automatic speech recognition (ASR) or target speech extraction.

Summary updated

Abstract

In recent years, there have been studies to further improve the end-to-end neural speaker diarization (EEND) systems. This letter proposes the EEND-DEMUX model, a novel framework utilizing demultiplexed speaker embeddings. In this work, we focus on disentangling speaker-relevant information in the latent space and then transform each separated latent variable into its corresponding speech activity. EEND-DEMUX can directly obtain separated speaker embeddings through the demultiplexing operation in the inference phase without an external speaker diarization system, an embedding extractor, or a heuristic decoding technique. Furthermore, we employ a multi-head cross-attention mechanism to capture the correlation between mixture and separated speaker embeddings effectively. We formulate three loss functions based on matching, orthogonality, and sparsity constraints to learn robust demultiplexed speaker embeddings. The experimental results on the LibriMix dataset show consistently improved performance in both a fixed and flexible number of speakers scenarios.
BibTeX
@misc{mun2023eend,
  author = {Sung Hwan Mun and Min Hyun Han and Chanyeong Moon and Nam Soo Kim},
  title = {{EEND-DEMUX: End-to-End Neural Speaker Diarization via Demultiplexed Speaker Embeddings}},
  year = {2023},
  eprint = {2312.06065},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2312.06065},
}
Figure