The DKU-DukeECE Diarization System for the VoxCeleb Speaker Recognition Challenge 2022

Weiqing Wang, Xiaoyi Qin, Ming Cheng, Yucong Zhang, Kangyue Wang, Ming LiarXiv 2022 · DOI · arXiv

Abstract

This paper describes the DKU-DukeECE submission to the 4th track of the VoxCeleb Speaker Recognition Challenge 2022 (VoxSRC-22). Our system contains a fused voice activity detection model, a clustering-based diarization model, and a target-speaker voice activity detection-based overlap detection model. Overall, the submitted system is similar to our previous year's system in VoxSRC-21. The difference is that we use a much better speaker embedding and a fused voice activity detection, which significantly improves the performance. Finally, we fuse 4 different systems using DOVER-lap and achieve 4.75 of the diarization error rate, which ranks the 1st place in track 4.

Introduction

Speaker diarization is the process of partitioning an audio stream into homogeneous segments according to speaker identity—essentially answering the question, "who spoke when?" While the field has matured, two major hurdles remain: overlapping speech and domain mismatch. Standard clustering-based systems often fail during overlapping segments, and models trained on clean data like VoxCeleb struggle with "in-the-wild" scenarios like VoxConverse. This work aims to push the boundaries of accuracy (measured by DER) by refining Voice Activity Detection (VAD) reliability and improving overlap handling.

Methods

The DKU-DukeECE architecture is a multi-stage pipeline emphasizing "fusion at every level."

Fused VAD

Instead of relying on one model, the team used four: a ResNet34-transformer, a Conformer-based model, the off-the-shelf pyannote.audio 2.0, and an ASR-based model. These were combined via majority voting, reducing the DER by 0.3% compared to single models.

Performance comparison of individual VAD models and their fusion.

Advanced Embeddings and Clustering

A SimAM-ResNet34 front-end with Attentive Statistics Pooling (ASP) extracted 256-dimensional speaker vectors. Pre-trained on VoxCeleb2, it was fine-tuned using pseudo-labeling to bridge the domain gap. For clustering, Agglomerative Hierarchical Clustering (AHC) and Spectral Clustering (SC) were compared. AHC provided lower DER, while SC was better at estimating the number of speakers.

TS-VAD for Overlap

To handle overlapping speech, Target-Speaker Voice Activity Detection (TS-VAD) was utilized. Unlike simple overlap detection, TS-VAD uses speaker embeddings to re-evaluate which specific speakers are active in a segment.

Datasets

The system was trained and validated using:

  • VoxCeleb 1 & 2: For core speaker embedding models.
  • VoxConverse (Dev/Test): For VAD training and hyper-parameter tuning.
  • LibriSpeech: To simulate synthetic overlapping speech for TS-VAD.
  • MUSAN & RIRs: For data augmentation (noise and reverberation).

Key Results and Insights

The system's performance was evaluated on the VoxSRC-22 challenge set. A major insight was the "Partial Assignment" strategy: replacing clustering output entirely with TS-VAD results (full assignment) increased speaker confusion, whereas only partially assigning overlap regions to AHC results led to a cleaner DER reduction.

Diarization performance (DER and JER) for various system configurations and fusion.

The final fused system using DOVER-lap achieved a 4.75% DER, securing 1st place in the competition. This was a significant improvement over the 19.60% baseline.

Conclusion

The DKU-DukeECE system demonstrates the power of multi-model fusion and sophisticated overlap handling. While AHC achieves the lowest DER, it tends to overestimate speaker counts, suggesting a need for hybrid clustering approaches in future work. Additionally, narrowing the domain gap between training data and real-world benchmarks remains a priority.

Summary updated

Abstract

This paper discribes the DKU-DukeECE submission to the 4th track of the VoxCeleb Speaker Recognition Challenge 2022 (VoxSRC-22). Our system contains a fused voice activity detection model, a clustering-based diarization model, and a target-speaker voice activity detection-based overlap detection model. Overall, the submitted system is similar to our previous year's system in VoxSRC-21. The difference is that we use a much better speaker embedding and a fused voice activity detection, which significantly improves the performance. Finally, we fuse 4 different systems using DOVER-lap and achieve 4.75 of the diarization error rate, which ranks the 1st place in track 4.
BibTeX
@misc{wang2022dku,
  author = {Weiqing Wang and Xiaoyi Qin and Ming Cheng and Yucong Zhang and Kangyue Wang and Ming Li},
  title = {{The DKU-DukeECE Diarization System for the VoxCeleb Speaker Recognition Challenge 2022}},
  year = {2022},
  eprint = {2210.01677},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.01677},
}
Figure