Abstract
This report describes the submission of the DKU-DukeECE-Lenovo team to the VoxCeleb Speaker Recognition Challenge (VoxSRC) 2021 track 4. Our system includes a voice activity detection (VAD) model, a speaker embedding model, two clustering-based speaker diarization systems with different similarity measurements, two different overlapped speech detection (OSD) models, and a target-speaker voice activity detection (TS-VAD) model. Our final submission, consisting of 5 independent systems, achieves a DER of 5.07% on the challenge test set.
Introduction
Speaker diarization—the task of determining "who spoke when"—remains a significant challenge in multi-speaker audio processing. While advancements in speaker embeddings have improved identity verification, two primary obstacles persist: handling overlapping speech and maintaining robustness across diverse acoustic environments. The VoxConverse dataset, used in the VoxSRC 2021 challenge, exemplifies these difficulties with varying noise levels, reverberation, and complex conversational dynamics. This paper details a multi-stage pipeline designed to address these hurdles through robust voice activity detection and neural refinement techniques.
Methods and System Architecture
Voice Activity Detection (VAD)
The system utilizes a ResNet34 front-end with BiLSTM layers. To enhance robustness, the team implemented a multi-resolution approach, training four models with different temporal resolutions (pooling scales S=1, 2, 4, 8). The final VAD output is a fusion of these models, which significantly reduces false alarms and missed speech segments.
Speaker Embeddings and Clustering
Speaker features are extracted using a ResNet34 architecture trained with ArcFace loss. For partitioning, two distinct clustering methods were employed: standard Agglomerative Hierarchical Clustering (AHC) and Spectral Clustering enhanced by an LSTM-based similarity measurement.
Neural Refinement: TS-VAD and Overlap Detection
A critical component of the system is the Target-Speaker VAD (TS-VAD) refinement stage. This model re-scans audio using initial speaker embeddings to precisely locate active regions for specific speakers. To specifically address overlapping speech, the authors proposed a "2-speaker TS-VAD" variant. By feeding pairs of speaker embeddings into the model, the system can detect overlaps between specific individuals, making it more flexible for recordings with varying speaker counts.
Datasets and Training
The models were trained on a diverse set of corpora, including VoxCeleb 1 & 2 for embeddings and AMI, ICSI, and DIHARD II & III for VAD and clustering. To pre-train the TS-VAD models, 900 hours of multi-speaker audio were simulated using Librispeech. Data augmentation techniques, such as MUSAN noise injection and RIR-based reverberation, were applied to improve generalizability to the VoxConverse test set.
Key Results
The system's performance was evaluated using the Diarization Error Rate (DER) and Jaccard Error Rate (JER). The final fusion of five independent systems, combined via DOVER-Lap, achieved a DER of 5.07% on the VoxSRC21 test set, a substantial improvement over the 17.99% baseline.
Ablation studies using "Oracle VAD" (perfect voice detection) indicated a theoretical DER limit of 2.61% for the clustering components, suggesting that VAD errors remain a primary contributor to the remaining error gap.
Conclusion
The DKU-DukeECE-Lenovo system demonstrates that combining multi-resolution VAD with neural refinement (TS-VAD) and system fusion can effectively tackle the complexities of real-world speaker diarization. Future work will focus on reducing the inference latency of the TS-VAD component and improving the robustness of the initial clustering stage to ensure no speakers are missed before the refinement phase.
Summary updated