TL;DR
This paper details the winning systems for the DIHARD Speech Diarization Challenge, showcasing a robust approach combining x-vector embeddings with Bayesian Hidden Markov Models (BHMM) for single-channel diarization. For multi-channel data, they achieved top performance by fusing channel-specific PLDA scores before agglomerative hierarchical clustering.
What Problem Does This Paper Address?
Speaker diarization, the task of identifying "who spoke when," is crucial for many speech processing applications. However, real-world speech is often noisy, reverberant, and can involve overlapping speakers, making accurate diarization a significant challenge. The DIHARD (DIstributional-HETERogeneous Audio REcordings DIarization) challenge specifically aims to push the boundaries of diarization systems on difficult, diverse audio data, including both single-channel and multi-channel recordings, with and without oracle voice activity detection (VAD) labels. This paper addresses the need for high-performance diarization systems that can effectively handle these challenging conditions.
Methods
The BUT team's winning systems for the DIHARD challenge employed a multi-stage approach. For tracks 1 and 2 (single-channel), the core pipeline involved:
- Signal Pre-processing: Weighted Prediction Error (WPE) was used to effectively remove late reverberation from the audio signals.
- x-vector Extraction: DNN-based speaker embeddings (x-vectors) were extracted at a higher frame rate (0.25s) than typical to capture finer temporal details.
- Agglomerative Hierarchical Clustering (AHC) Initial Clustering: x-vectors were initially clustered using AHC, with similarity measured by Probabilistic Linear Discriminant Analysis (PLDA) log-likelihood ratios. Two PLDA models were trained: one on out-of-domain VoxCeleb data and another adapted to the DIHARD development data. An interpolated PLDA model was used for better performance.
- Bayesian HMM (BHMM) for x-vector Clustering: This was the core diarization step. BHMM, also known as VB diarization, was used to refine the x-vector clusters. This method uses Variational Bayes inference to iteratively update speaker assignments and automatically determine the number of speakers.
- BHMM-based Re-segmentation: An additional frame-level VB re-segmentation step using MFCCs and another BHMM was applied to further refine speaker boundaries.
- Overlapped Speech Post-processing: A heuristic was developed to handle overlapping speech by assigning frames to the two closest speakers in time.
For tracks 3 and 4 (multi-channel), a simpler approach was adopted:
- Multi-channel AHC Clustering: The primary method involved extracting x-vectors from each channel independently. Then, pairwise PLDA score matrices from each channel were averaged, and AHC was performed on this fused score matrix. Beamforming was also explored but did not yield better results.
- Voice Activity Detection (VAD): For track 4, where oracle VAD labels were not provided, a DNN-based VAD system was trained and used.
Datasets
- DIHARD Dataset: Used for both training and evaluation across all four tracks. This dataset is characterized by its challenging nature, including noise and reverberation.
- CHiME Challenge Data: Used for tracks 3 and 4, featuring multi-channel recordings.
- VoxCeleb 1 and 2: Used for training the x-vector extractor and PLDA models.
Key Insights and Ideas
- The Power of BHMM: The paper strongly advocates for BHMM over simpler AHC for x-vector clustering, demonstrating significant improvements in diarization error rate (DER). BHMM's ability to iteratively refine speaker assignments and automatically determine speaker counts is key.
- Domain Adaptation is Crucial: Adapting PLDA models to the target domain (DIHARD data) significantly improves performance compared to using models trained solely on out-of-domain data (VoxCeleb).
- Overlapped Speech is a Major Hurdle: The authors highlight that overlapped speech accounts for a substantial portion of the DER, underscoring the need for more sophisticated methods to handle it. Their post-processing heuristic provided some improvement but is not a complete solution.
- Multi-channel Fusion Strategy: For multi-channel data, fusing channel-specific PLDA scores before clustering proved more effective than processing individual channels or using beamforming.
- Higher x-vector Frame Rate Benefits: Extracting x-vectors at a higher frame rate (0.25s) was found to improve diarization results, likely due to better temporal resolution.
Key Results
- Winning Performance: The BUT team secured first place in all four tracks of the Second DIHARD Speech Diarization Challenge.
- BHMM vs. AHC: Using BHMM for x-vector clustering improved DER by approximately 2% absolute compared to AHC alone on the DIHARD dataset.
- Re-segmentation Gains: The frame-level re-segmentation step provided marginal gains (around 0.2-0.3% DER improvement) when combined with the advanced BHMM x-vector clustering, likely due to the already high quality of the BHMM output and the higher x-vector frame rate.
- Overlapped Speech Impact: The overlapped speech post-processing step reduced DER by about 0.2% on the evaluation set.
- Multi-channel Fusion: Fusion of PLDA score matrices from individual channels achieved the best result for multi-channel diarization, with a DER of 47.93% on the evaluation set, outperforming single-channel approaches.
Limitations and Future Work
- Overlapped Speech Handling: The current methods for handling overlapped speech are heuristic and do not fully resolve the issue, contributing significantly to the overall DER. Future work should focus on more advanced techniques for this.
- VAD System for Multi-channel: For track 4, a tailored VAD system trained on the multi-channel training data is suggested as future work to improve performance when oracle VAD labels are unavailable.
- Beamforming Optimization: The beamforming approach for multi-channel data did not yield optimal results, and tuning its parameters is identified as future work.
- PLDA Adaptation for Re-segmentation: Adapting the re-segmentation BHMM (specifically, the i-vector model) to the target domain is suggested as a way to improve re-segmentation results.
- General x-vector Adaptation: The authors believe that further research into adapting x-vectors to in-domain data is a promising direction.
Summary updated