Abstract
In this paper, we present the submitted system for the third DIHARD Speech Diarization Challenge from the DKU-Duke-Lenovo team. Our system consists of several modules: voice activity detection (VAD), segmentation, speaker embedding extraction, attentive similarity scoring, and agglomerative hierarchical clustering. In addition, target speaker VAD (TSVAD) is used for phone call data to further improve performance. Our final submitted system achieves a DER of 15.43% for the core evaluation set and 13.39% for the full evaluation set on Task 1, and a DER of 21.63% for the core evaluation set and 18.90% for the full evaluation set on Task 2.
Introduction
Speaker diarization—the process of determining "who spoke when"—is notoriously difficult in "wild" conditions. While standard systems work well in clean environments, they often struggle with the DIHARD dataset, which includes noisy backgrounds, diverse recording equipment, and critically, overlapping speech. Traditional systems typically fail to assign a single time frame to multiple speakers simultaneously. The DKU-Duke-Lenovo team addressed these challenges by combining robust deep-learning-based feature extraction with specialized post-processing techniques, specifically targeting overlapping speech scenarios.
Methodology
The system utilizes a bifurcated pipeline based on audio type: Conversation Telephone Speech (CTS) and Non-Conversation Telephone Speech (NCTS). This classification is performed by analyzing frequency content, as CTS data typically involves two speakers and specific acoustic properties.
Voice Activity Detection and Speaker Embeddings
The team employed a ResNet18 combined with a 2-layer Bi-LSTM for Voice Activity Detection (VAD) to distinguish speech from silence and noise. For speaker representation, a ResNet34 architecture trained with ArcFace loss was used to extract highly discriminative "speaker fingerprints."
Target-Speaker Voice Activity Detection (TSVAD)
The core innovation for handling overlapping speech in CTS data is the Target-Speaker Voice Activity Detection (TSVAD). Unlike standard VAD, TSVAD takes a specific speaker's embedding as an additional input and predicts that specific person's activity frame-by-frame.
This architecture allows the system to detect when both speakers are talking at once. Instead of feeding raw acoustic features, the system uses a ResNet34 to extract frame-level "identity vectors," which are then processed through Bi-LSTM layers to find relationships between speakers efficiently.
Datasets
To ensure robustness, the system was trained on a massive array of datasets, including VoxCeleb 1 & 2 for embeddings, the AMI and ICSI meeting corpora for similarity measurement, and MUSAN/RIRS for data augmentation. CTS-specific training utilized SRE (2004-2008) databases and Switchboard.
Key Results and Insights
The most significant performance gains came from treating CTS and NCTS differently. Because CTS is limited to two speakers, the team could apply more aggressive overlap detection (TSVAD) that might be too complex for multi-speaker NCTS data.
As shown in the experimental results, applying TSVAD on the CTS development set reduced the Diarization Error Rate (DER) from 15.07% to 7.63% after two rounds of refinement. This confirms that frame-level speaker detection is far superior to segment-level clustering for handling overlaps. The final challenge results for the combined system are summarized below:
| Task | Full Evaluation Set (DER) | Core Evaluation Set (DER) |
|---|---|---|
| Task 1 (Labels provided) | 13.39% | 15.43% |
| Task 2 (No labels) | 18.90% | 21.63% |
Conclusion
The DKU-Duke-Lenovo system demonstrates that specialized pipelines and frame-level detection (TSVAD) are essential for high-performance diarization in complex environments. While TSVAD worked exceptionally well for telephone speech, it performed less effectively on NCTS data due to extreme domain diversity and high noise levels. Future work will focus on training domain-specific models and adapting TSVAD for scenarios involving more than two speakers.
Summary updated