Speaker Diarization with Deep Speaker Embeddings for DIHARD Challenge II

Sergey Novoselov, Aleksei Gusev, Artem Ivanov, Timur Pekhovsky, Andrey Shulipa, Anastasia Avdeeva, Artem Gorlanov, Alexandr KozlovInterspeech 2019

TL;DR

This paper presents the ITMO University team's approach to the DIHARD Challenge II, focusing on speaker diarization in challenging, noisy, and far-field audio conditions. They leverage advanced deep neural network (DNN) speaker embeddings (x-vectors and c-vectors) and explore novel clustering and scoring techniques, including a discriminative cosine similarity metric learning (CSML) approach and an environment-aware adaptive thresholding strategy for agglomerative hierarchical clustering (AHC).

What Problem Does This Paper Address?

Speaker diarization, the task of segmenting an audio stream into segments belonging to different speakers, is crucial for many voice-based services. However, real-world audio often presents significant challenges: distant microphones, low signal-to-noise ratios (SNR), and overlapping speech. The DIHARD (DIstributed HARD) challenges specifically aim to push the boundaries of diarization systems by providing datasets that mimic these difficult conditions. This paper addresses the need for robust speaker diarization systems that can perform effectively under such adverse acoustic environments, building upon the success of deep learning-based speaker embeddings seen in previous challenges.

Methods

The core of the ITMO team's approach relies on deep speaker embeddings. They explore two main types:

  • x-vectors: These are standard DNN-based speaker embeddings, extracted from the statistics pooling layer of a classification network. The systems are largely based on established configurations, with some modifications.
  • c-vectors: These are a more recent type of embedding, based on Speaker Residual Net architectures, which the authors also investigate.

For clustering these embeddings, they primarily use Agglomerative Hierarchical Clustering (AHC). Instead of relying solely on traditional Probabilistic Linear Discriminant Analysis (PLDA) for scoring within AHC, they introduce and evaluate a Cosine Similarity Metric Learning (CSML) model. This discriminative approach aims to learn a transformation matrix that optimizes cosine similarities between embeddings, potentially offering better performance in noisy conditions.

A key innovation is the Recording Condition Detector (RCD). This component is trained to classify the acoustic environment of a given audio segment. Based on this classification, the system dynamically adjusts the AHC threshold. This adaptive thresholding strategy is hypothesized to improve diarization accuracy by tailoring the clustering process to the specific speech quality.

Finally, the paper mentions system fusion using an ensemble of decision trees, combining scores from different systems to achieve better overall performance.

Datasets

The paper focuses on the DIHARD Challenge II dataset. This dataset is characterized by challenging recording conditions, including far-field microphones, low SNR, and overlapping speech. The authors mention training their RCD on a development set and augmenting it with data from standard sources like MUSAN and RIR datasets to simulate various distortions. The training data for the speaker embeddings themselves includes large corpora like VoxCeleb1, VoxCeleb2, and SITW, along with their augmented versions, totaling approximately 5.2 million examples from 7562 speakers.

Key Insights and Ideas

  • Deep Embeddings are Key: The paper reinforces the idea that advanced deep neural network embeddings (x-vectors and c-vectors) are essential for robust speaker diarization in challenging conditions, significantly outperforming older feature-based methods.
  • Discriminative Scoring Matters: The exploration of CSML as an alternative to PLDA for scoring within AHC suggests that discriminative approaches can be competitive or even superior in certain scenarios, especially when dealing with noisy data.
  • Adaptive Thresholding is Crucial: The RCD and adaptive AHC thresholding strategy is a significant finding. It highlights that a one-size-fits-all clustering threshold is suboptimal for diverse acoustic conditions, and tailoring it based on detected environmental characteristics can yield substantial improvements.
  • Fusion Benefits: Combining multiple systems, even those based on similar underlying technologies, can lead to further performance gains, demonstrating the robustness gained from diverse approaches.

Key Results

The paper presents results in terms of Diarization Error Rate (DER) on the development and evaluation sets.

  • The baseline x-vector system with PLDA scoring achieved a DER of 23.95% on the development set.
  • A c-vector based system (SO) performed slightly worse at 24.10% DER.
  • More advanced x-vector configurations, particularly the Xvec-Ext-TDNN-LSTM system, showed significant improvements, reaching 22.24% DER on the development set.
  • The fusion of x-vector and c-vector systems (e.g., S3 vs. S4) demonstrated a 1% DER improvement on the evaluation set.
  • The adaptive AHC thresholding scheme, as seen in systems S5-S9, resulted in an almost 2% lower DER compared to systems using a common threshold (S3-S4), highlighting its effectiveness.

Limitations and Future Work

  • C-vector Performance: The paper notes that their initial c-vector system (SO) was slightly inferior to the baseline x-vector system. This suggests further research might be needed to optimize c-vector extraction or training for diarization.
  • Fusion Complexity: While fusion improved results, the paper doesn't delve deeply into the computational cost or complexity of the fusion mechanism itself.
  • Single Channel Focus: The presented results are specifically for the single-channel task with reference SAD. Future work could extend these findings to multi-channel scenarios and systems using automatically detected speech activity.
  • Further Exploration of Embeddings: The authors explored specific DNN architectures. There's always room to investigate even newer or more specialized embedding extraction techniques.
  • RCD Robustness: While the RCD shows promise, its performance across a wider range of unseen acoustic conditions would be a valuable area for future study.

Summary updated

BibTeX
@inproceedings{novoselov2019speaker,
  author = {Sergey Novoselov and Aleksei Gusev and Artem Ivanov and Timur Pekhovsky and Andrey Shulipa and Anastasia Avdeeva and Artem Gorlanov and Alexandr Kozlov},
  title = {{Speaker Diarization with Deep Speaker Embeddings for DIHARD Challenge II}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure