Abstract
This paper describes the DKU-DukeECE submission to the 4th track of the VoxCeleb Speaker Recognition Challenge 2022 (VoxSRC-22). Our system contains a fused voice activity detection model, a clustering-based diarization model, and a target-speaker voice activity detection-based overlap detection model. Overall, the submitted system is similar to our previous year's system in VoxSRC-21. The difference is that we use a much better speaker embedding and a fused voice activity detection, which significantly improves the performance. Finally, we fuse 4 different systems using DOVER-lap and achieve 4.75 of the diarization error rate, which ranks the 1st place in track 4.
Introduction
Speaker diarization is the process of partitioning an audio stream into homogeneous segments according to speaker identity—essentially answering the question, "who spoke when?" While the field has matured, two major hurdles remain: overlapping speech and domain mismatch. Standard clustering-based systems often fail during overlapping segments, and models trained on clean data like VoxCeleb struggle with "in-the-wild" scenarios like VoxConverse. This work aims to push the boundaries of accuracy (measured by DER) by refining Voice Activity Detection (VAD) reliability and improving overlap handling.
Methods
The DKU-DukeECE architecture is a multi-stage pipeline emphasizing "fusion at every level."
Fused VAD
Instead of relying on one model, the team used four: a ResNet34-transformer, a Conformer-based model, the off-the-shelf pyannote.audio 2.0, and an ASR-based model. These were combined via majority voting, reducing the DER by 0.3% compared to single models.
Advanced Embeddings and Clustering
A SimAM-ResNet34 front-end with Attentive Statistics Pooling (ASP) extracted 256-dimensional speaker vectors. Pre-trained on VoxCeleb2, it was fine-tuned using pseudo-labeling to bridge the domain gap. For clustering, Agglomerative Hierarchical Clustering (AHC) and Spectral Clustering (SC) were compared. AHC provided lower DER, while SC was better at estimating the number of speakers.
TS-VAD for Overlap
To handle overlapping speech, Target-Speaker Voice Activity Detection (TS-VAD) was utilized. Unlike simple overlap detection, TS-VAD uses speaker embeddings to re-evaluate which specific speakers are active in a segment.
Datasets
The system was trained and validated using:
- VoxCeleb 1 & 2: For core speaker embedding models.
- VoxConverse (Dev/Test): For VAD training and hyper-parameter tuning.
- LibriSpeech: To simulate synthetic overlapping speech for TS-VAD.
- MUSAN & RIRs: For data augmentation (noise and reverberation).
Key Results and Insights
The system's performance was evaluated on the VoxSRC-22 challenge set. A major insight was the "Partial Assignment" strategy: replacing clustering output entirely with TS-VAD results (full assignment) increased speaker confusion, whereas only partially assigning overlap regions to AHC results led to a cleaner DER reduction.
The final fused system using DOVER-lap achieved a 4.75% DER, securing 1st place in the competition. This was a significant improvement over the 19.60% baseline.
Conclusion
The DKU-DukeECE system demonstrates the power of multi-model fusion and sophisticated overlap handling. While AHC achieves the lowest DER, it tends to overestimate speaker counts, suggesting a need for hybrid clustering approaches in future work. Additionally, narrowing the domain gap between training data and real-world benchmarks remains a priority.
Summary updated