UWB-NTIS Speaker Diarization System for the DIHARD II 2019 Challenge

Zbyněk Zajíc, Marie Kunešová, Marek Hrúz, Jan VaněkInterspeech 2019

TL;DR

This paper presents a speaker diarization system for the DIHARD II challenge, improving upon previous methods by combining i-vectors and x-vectors into "xi-vectors" and employing a novel domain classifier to adapt system parameters. Their system achieved a Diarization Error Rate (DER) of 23.47% on the evaluation set, demonstrating the effectiveness of their approach.

What Problem Does This Paper Address?

Speaker diarization, often referred to as "who spoke when," is a crucial task in speech processing. It aims to segment an audio recording and label each segment with the identity of the speaker. This is challenging due to variations in speech, background noise, overlapping speech, and the unknown number of speakers in a given recording. The DIHARD II challenge specifically focuses on these difficulties by providing diverse datasets with varying acoustic conditions and speaker characteristics. The authors aim to build a robust speaker diarization system that can perform well across these challenging conditions, particularly for the DIHARD II challenge's Track 1, which uses reference speech activity detection (SAD).

Methods

The core of the UWB-NTIS system follows a standard speaker diarization pipeline: segmentation, i-vector/x-vector extraction, clustering, and resegmentation.

  • Segmentation: The system first splits audio into speech regions. A key novelty is the use of a Speaker Change Detector (SCD-CNN), a Convolutional Neural Network trained to predict speaker changes, which refines the initial segmentation. Segments are also constrained to a minimum duration of 0.5 seconds.
  • Feature Extraction: Linear Frequency Cepstral Coefficients (LFCCs) are used as the primary acoustic features, with Cepstral Mean Normalization (CMN) applied for channel compensation.
  • Segment Description (xi-vectors): A significant contribution is the creation of "xi-vectors" by concatenating i-vectors and x-vectors. i-vectors are traditional speaker embeddings, while x-vectors are more recent DNN-based embeddings. This fusion aims to leverage the strengths of both. Longer segments are further split to ensure sufficient data for vector extraction.
  • Clustering: Agglomerative Hierarchical Clustering (AHC) is the primary clustering method, using the cosine distance between xi-vectors. The system also incorporates domain-specific knowledge to set minimum and maximum cluster counts. For specific corpora with a high prevalence of two-speaker conversations, Probabilistic Linear Discriminant Analysis (PLDA) is used for distance evaluation, followed by k-medoids clustering.
  • Resegmentation: To refine segment boundaries, a Gaussian Mixture Model (GMM) is computed for each speaker cluster, and the entire conversation is reclassified frame-by-frame based on GMM likelihoods.
  • Domain Classification: A novel hierarchical two-stage neural network classifier is introduced. The first stage distinguishes between single-speaker and multi-speaker recordings. The second stage identifies the specific corpus (out of 11 in the DIHARD II development set) or classifies it as "unknown." This classifier uses i-vectors extracted from the entire conversation and allows for domain-specific tuning of system parameters, particularly the AHC threshold.

Datasets

  • Training Data: LibriSpeech, VoxCeleb2, TedLium3, and ELRA corpora (Speecon database UK English, US English). Data augmentation (additive noise, music, babble, reverberation) was applied to LibriSpeech and TedLium3.
  • Development Data: DIHARD II development set (comprising 11 corpora).
  • Evaluation Data: DIHARD II evaluation set (Track 1).

The DIHARD II corpus is noted for its diversity, including different domains with varying numbers of speakers, noise levels, and audio quality.

Key Insights and Ideas

  • Xi-vectors for improved representation: Combining i-vectors and x-vectors into "xi-vectors" proved beneficial, likely capturing a richer representation of speaker characteristics than either alone.
  • Domain adaptation is crucial: The domain classifier significantly improves performance by allowing the system to adapt its configuration (e.g., clustering thresholds) to the specific characteristics of different corpora within the DIHARD II dataset. This highlights the importance of understanding the data's origin.
  • SCD-CNN for better segmentation: The Convolutional Neural Network-based Speaker Change Detector (SCD-CNN) outperformed Kaldi's SAD segmentation in terms of purity, meaning its segments were more likely to contain only one speaker. This is critical for accurate diarization.
  • Late system combination: Using the Kaldi system for "unknown domain" recordings and for specific problematic corpora (Seedlings, VAST) demonstrated a practical approach to leveraging external systems when the primary system's performance might be less reliable.

Key Results

  • The best submission achieved a Diarization Error Rate (DER) of 23.47% and a Jaccard Error Rate (JER) of 48.99% on the DIHARD II evaluation set (Track 1).
  • The SCD-CNN segmentation method showed higher purity compared to Kaldi's SAD segmentation across most corpora in the development set.
  • The domain classifier achieved 100% accuracy for the first stage (single vs. multi-speaker) and 82% accuracy for the second stage (corpus identification) on the development data.
  • A comparison of segment descriptors (i-vec, x-vec, xi-vec) showed that xi-vectors yielded the lowest DER (22.51%) for their system on the development set.

Limitations and Future Work

  • Overlap Detection: The authors acknowledge that they were unable to train a real overlap detector with sufficient accuracy for the DIHARD II data, despite recognizing its potential benefits.
  • De-noising: While de-noising was investigated, it was omitted from the final system due to the lack of clear benefits in their experiments.
  • Future work could focus on: Developing more robust overlap detection mechanisms and exploring more advanced speech enhancement techniques that are effective for diarization. The paper also implies that further investigation into the fusion of different embedding types and more sophisticated domain adaptation strategies could yield further improvements.

Summary updated

BibTeX
@inproceedings{zajic2019uwb,
  author = {Zbyněk Zajíc and Marie Kunešová and Marek Hrúz and Jan Vaněk},
  title = {{UWB-NTIS Speaker Diarization System for the DIHARD II 2019 Challenge}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure