Abstract
EEND-DEMUX is a novel end-to-end neural speaker diarization model designed to improve "who spoke when" detection by disentangling speaker-specific information directly in the latent embedding space. By employing a "demultiplexing" operation alongside specific structural constraints—matching, orthogonality, and sparsity—the model achieves significantly lower error rates than previous state-of-the-art methods on both fixed and flexible numbers of speakers.
Introduction
Speaker diarization is the process of partitioning an audio stream into segments based on speaker identity. Traditional systems typically rely on clustering-based pipelines involving separate modules for voice activity detection, segmentation, and embedding extraction. These systems often fail during "overlapping speech"—when multiple individuals speak simultaneously—because they usually assign a time segment to only one person.
End-to-End Neural Diarization (EEND) was developed to address this by predicting multi-speaker activity directly. However, standard EEND models often struggle with a flexible number of speakers or require complex auxiliary tasks like full speech separation. EEND-DEMUX bridges this gap by disentangling speakers in the neural network's internal representation without the overhead of full audio separation.
Methods: The EEND-DEMUX Architecture
The EEND-DEMUX model consists of four main components designed to extract and separate speaker information efficiently:
- MixtureEncoder: A Transformer-based encoder that processes raw audio features into "mixture embeddings" containing information from all speakers.
- Demultiplexer: The core novelty of the system. It uses parallel 1D-CNN modules to split the mixture embedding into $S$ separate "demultiplexed" speaker embeddings at every frame.
- AttractorDecoder: This module uses Multi-Head Cross-Attention (MHCA) to find "attractors" (representative vectors for each speaker) by analyzing the correlation between the mixture and the separated embeddings.
- SpeakerEncoder: A pre-trained network that provides "oracle" speaker embeddings during training to guide the demultiplexing process.
Structural Constraints
To optimize the demultiplexing process, the authors introduce a total loss function with three unique constraints:
- Matching (Distillation): Forces the demultiplexed embeddings to align with the "clean" oracle embeddings.
- Orthogonality: Ensures embeddings from different speakers are mathematically distinct while embeddings from the same speaker remain similar.
- Sparsity: Uses an $\ell_1$-norm penalty to encourage the model to focus on the most important features, reducing overfitting.
Key Results
The EEND-DEMUX model was evaluated using the LibriMix dataset, a standard benchmark for overlapping speech. The model consistently outperformed baseline EEND models across all scenarios.
As shown in the results for the 2-speaker scenario, the combination of structural constraints leads to a significant reduction in the Diarization Error Rate (DER). Key performance highlights include:
- 2-Speaker Scenario: Achieved a DER of 3.79%, a relative improvement of 24.50% over the baseline.
- 3-Speaker Scenario: Achieved a DER of 4.91%, a relative improvement of 18.17%.
- Flexible Speakers: On the combined Libri2&3Mix dataset, the model achieved a DER of 4.39%, outperforming state-of-the-art models like TS-VAD and USED.
Conclusion
EEND-DEMUX demonstrates that disentangling speakers in the embedding space is a highly effective strategy for end-to-end diarization. By avoiding the computational cost of full signal reconstruction while maintaining the benefits of multi-task learning, the model sets a new standard for handling overlapping speech. Future work will explore integrating this framework into larger multi-talker systems, such as automatic speech recognition (ASR) or target speech extraction.
Summary updated