Abstract
This paper presents NSD-MS2S-SSMoE, a state-of-the-art speaker diarization system designed for challenging "in-the-wild" acoustic environments. By integrating a Shared and Soft Mixture of Experts (SS-MoE) module and a Deep Interactive Module (DIM) within a sequence-to-sequence framework, the system effectively handles overlapping speech and domain mismatch. The proposed approach achieved first place in the CHiME-7 challenge, demonstrating significant error reductions across diverse datasets like CHiME-6, DiPCo, and DIHARD-III.
Introduction
Speaker diarization—identifying "who spoke when"—remains a significant challenge in real-world scenarios characterized by noise, reverberation, and overlapping speech. Traditional clustering methods often fail during overlaps, while existing neural approaches like Target-Speaker Voice Activity Detection (TS-VAD) can be computationally expensive and prone to domain bias. Furthermore, noisy initial speaker embeddings can degrade the entire diarization pipeline. This work addresses these issues by proposing a robust, efficient architecture that leverages Mixture of Experts (MoE) to adapt to varied acoustic conditions.
Methodology
The core of the proposed system is the NSD-MS2S-SSMoE architecture, which combines memory-aware speaker embeddings with a Sequence-to-Sequence (Seq2Seq) framework.
Key components include:
- Shared and Soft Mixture of Experts (SS-MoE): Instead of standard static feed-forward layers, the decoder utilizes an SS-MoE module. It features a "Shared Expert" for general patterns and multiple "Collaborative Experts" for specialized acoustic nuances. A differentiable "Soft Routing" algorithm assigns weights to these experts, enabling dynamic adaptation to different environments. The authors found that placing these modules in the later stages of the decoder (layers 4–6) yielded the best performance.
- Deep Interactive Module (DIM): To improve embedding robustness, the DIM replaces simple additive attention in the memory module with a multi-layer dot-product attention mechanism. This allows the model to "interrogate" a memory bank of speaker basis vectors to extract cleaner, more discriminative embeddings even from degraded audio.
- Parameter Transfer Strategy: To mitigate the high training cost of MoE models, the authors developed a method to initialize experts using weights from a pre-trained non-MoE model, reducing retraining costs by over 50%.
Results
The system was evaluated on CHiME-6, DiPCo, Mixer 6, and DIHARD-III datasets, utilizing approximately 5,300 hours of audio for training.
- CHiME-7 Champion: The system secured first place in the main track of the CHiME-7 challenge.
- Error Reduction: The MoE-enhanced model achieved a 3.19% relative reduction in Diarization Error Rate (DER) on CHiME-6 and a 6.39% reduction on the DiPCo development set.
- Embedding Quality: The DIM module specifically reduced the Speaker Error (SPKERR) component by 27% on the DiPCo dataset, confirming its ability to extract accurate speaker identities from noisy audio.
- Generalization: The system achieved state-of-the-art results on DIHARD-III, proving its effectiveness across diverse domains including broadcast news, courtroom audio, and clinical meetings.
Conclusion
The integration of SS-MoE and DIM significantly enhances the robustness and efficiency of speaker diarization in complex environments. By utilizing Mixture of Experts as a bias mitigator and employing a strategic parameter transfer strategy, the authors have developed a high-performing system that generalizes well across mismatched domains. Future work will focus on addressing potential overfitting on specific datasets and exploring how this MoE approach scales with even larger pre-trained speech models.
Summary updated