Introduction
Speaker diarization—the task of determining "who spoke when"—is a critical component in speech processing pipelines, particularly for multi-speaker environments. The LEAP team from the Indian Institute of Science participated in the Second DIHARD Challenge, a benchmark specifically designed to test systems against the most difficult real-world audio conditions. By introducing novel posterior scaling for Variational Bayes (VB-HMM) refinement and domain mismatch compensation for i-vector embeddings, the team achieved a significant 13.7% relative improvement in Diarization Error Rate (DER) over the challenge baseline.
The DIHARD Challenge: Beyond the Forgiveness Collar
Unlike earlier benchmarks that allowed for a "forgiveness collar" (a small time buffer around speaker changes) and often ignored overlapping speech, the DIHARD challenges demand high precision. The dataset includes chaotic, diverse audio from noisy restaurants, clinical consultations, child language acquisition recordings, and YouTube videos.
The authors identified two primary bottlenecks in standard pipelines:
- Model Sensitivity: Refinement models like the VB-HMM often struggle with "spurious speaker turns," where the system switches between speakers too rapidly or fails to distinguish them in high-noise conditions.
- Domain Gap: There is a significant mismatch between the clean, massive datasets used to train speaker embeddings (like VoxCeleb) and the messy, in-domain audio found in the DIHARD evaluation sets.
The LEAP Pipeline
The LEAP system utilizes a multi-stage pipeline that combines two distinct types of speaker representations: traditional statistical i-vectors and deep learning-based x-vectors.
As shown in the architecture above, the system processes audio through parallel feature extraction paths. The x-vector system utilizes Time Delay Neural Networks (TDNN), while the i-vector system incorporates Domain Mismatch Compensation (DMC). After PLDA scoring, the systems are fused via weighted average similarity scores before undergoing Agglomerative Hierarchical Clustering (AHC) and final refinement.
Core Innovations
1. Posterior Scaled VB-HMM
Standard systems use AHC for an initial guess, followed by VB-HMM refinement. The LEAP team proposed scaling the zeroth-order statistics—the frame-level speaker assignments—during the model's learning process. By introducing a scaling factor ($\beta$), they "boost" the evidence for the dominant speaker in a segment. This makes the model's decisions more decisive and less prone to the "jitter" often caused by background noise.
2. Domain Mismatch Compensation (DMC)
To bridge the gap between training and evaluation data, the team applied a variance normalization technique to i-vector features. They calculated the variance difference between out-of-domain training data and in-domain development data to create a transformation matrix. This aligns the features so the Probabilistic Linear Discriminant Analysis (PLDA) scoring model is not confused by changes in the recording environment.
Experimental Results
The LEAP system demonstrated consistent improvements across both development and evaluation sets. The scaling factor $\beta$ acted as a powerful regularizer; while $\beta=1$ represents the standard VB-HMM, the team found $\beta=24$ to be the optimal "sweet spot" for balancing speaker separation against the risk of over-merging distinct speakers.
As detailed in the results table, the fused system reached a DER of 20.56% on the development set and 21.90% on the evaluation set. The system was particularly effective in the "VAST" (YouTube) and "Clinical" domains, where posterior scaling helped navigate highly variable audio quality.
Conclusion
The LEAP system proves that even as deep learning-based x-vectors become the industry standard, traditional i-vectors still provide valuable complementary information when properly compensated for domain shifts. While the current system relies on manually tuned hyperparameters, future work aims to make these scaling factors adaptive—automatically adjusting the model's "confidence" based on estimated noise levels or speaker density. Additionally, accurately diarizing overlapping speech remains a primary frontier for future iterations of the system.
Summary updated