The Second DIHARD Challenge: System Description for USC-SAIL Team

Tae Jin Park, Manoj Kumar, Nikolaos Flemotomos, Monisankha Pal, Raghuveer Peri, Rimita Lahiri, Panayiotis G. Georgiou, Shrikanth NarayananInterspeech 2019 · OpenReview

Abstract

In this paper, we describe components that form a part of USC-SAIL team’s submissions to Track 1 and Track 2 of the second DIHARD speaker diarization challenge. We describe each module in our speaker diarization pipeline and explain the rationale behind our choice of algorithms for each module, while comparing the Diarization Error Rate (DER) against different module combinations. We propose a clustering scheme based on spectral clustering that yields competitive performance. Moreover, we introduce an overlap detection scheme and a re-segmentation system for speaker diarization and investigate their performances using controlled and in-the-wild conditions. In addition, we describe the additional components that will be integrated to our speaker diarization system. To pursue the best performance, we compare our system with the state-of-the-art methods that are presented in the previous challenge and literature. We include preliminary results of our speaker diarization system on the evaluation data from the second DIHARD challenge.

Introduction

Speaker diarization—the task of determining "who spoke when"—is relatively solved for clean, telephony speech. However, the DIHARD challenge focuses on the "hard" cases: clinical interviews, restaurant conversations, and field recordings characterized by high noise, reverberation, and frequent overlapping speech.

Traditional systems often rely on Agglomerative Hierarchical Clustering (AHC) paired with Probabilistic Linear Discriminant Analysis (PLDA). While powerful, these methods are brittle when there is a significant "domain mismatch" between training and testing data. The USC-SAIL team aims to build a pipeline that is robust to these diverse, unpredictable acoustic environments.

Methods

The authors propose a modular pipeline with several sophisticated components designed to handle acoustic variability.

Modular Speaker Diarization Pipeline
  • Hybrid Embeddings: The system utilizes x-vectors (Time-Delay Neural Network based embeddings) and experiments with a Hybrid DNN-TVM model, which replaces the standard statistical pooling layer with a Total Variability Model to better capture speaker characteristics.
  • Embedding Denoising (CGAN): Rather than just denoising audio, the team uses a Conditional Generative Adversarial Network (CGAN) to denoise the x-vector embeddings themselves. The generator learns to map "noisy" x-vectors to "clean" ones in the latent space.
  • Spectral Clustering (SC): Instead of AHC, the team employs Spectral Clustering with a binary affinity matrix. They use Eigen Gap Analysis to automatically estimate the number of speakers in a recording, addressing a common pain point in diarization.
  • Overlap Detection & Re-segmentation: A DNN was trained specifically to identify segments where multiple people are talking simultaneously. This is followed by a Viterbi-based re-segmentation step using Gaussian Mixture Models (GMMs) to refine the final speaker boundaries.

Experimental Setup

The team curated a massive training set to ensure robustness:

  • Training: A combination of NIST SRE ('04-'08), Switchboard, and VoxCeleb 1 & 2, totaling roughly 200,000 utterances (8,200 hours) of speech.
  • Augmentation: Extensive data augmentation was used, including additive noise (DEMAND, REVERB datasets) and RIR convolution (RWCP, Aachen databases) to simulate messy environments.
  • Evaluation: The system was tested on the Second DIHARD challenge development and evaluation sets, which include diverse domains like "Child Language," "Web Video," and "Clinical."

Key Insights and Discussion

The PLDA Mismatch

The authors found that while PLDA/AHC works well when the training data matches the test data, it fails on DIHARD's diverse domains. Spectral Clustering with Cosine Similarity proved much more consistent across varying datasets.

SNR-Aware Processing

One of the most practical insights was that overlap detection and re-segmentation are double-edged swords. These modules actually hurt performance in low-SNR (noisy) sessions. Consequently, the team optimized an SNR threshold to only apply these refinements when the signal quality was high enough to be reliable.

Mean F-score vs SNR (dB)

Latent Space Denoising

By applying a GAN to the x-vector space, the team targeted the specific features used for clustering, potentially ignoring noise that doesn't interfere with speaker identity, which is more efficient than cleaning the raw waveform.

Key Results

The team evaluated their system on two tracks: Track 1 (using gold-standard speech activity detection) and Track 2 (diarization from scratch).

  • Track 1 (Best Performance): Their fusion of two different x-vector extractors (x-vec I+II) achieved a Diarization Error Rate (DER) of 21.82% on the Dev set and 22.89% on the Eval set.
  • Track 2: When the system had to detect speech itself (using their 3-layer MLP SAD), the DER rose to 46.72% on the Eval set, highlighting how much "who spoke when" depends on "is someone speaking at all."
  • Clustering Comparison: Spectral clustering consistently yielded lower DERs than AHC+PLDA when faced with the extreme variability of the DIHARD evaluation set.

Limitations and Future Work

A significant performance gap remains between the development and evaluation sets, suggesting that even with 8,200 hours of data, the system still struggled with the "unseen" acoustic conditions of the final challenge. Future work will focus on developing more aggressive data augmentation strategies, refining the binarization threshold for spectral clustering, and extending SNR-dependent processing to the embedding extractors themselves.

Summary updated

Abstract

In this paper, we describe components that form a part of USC-SAIL team’s submissions to Track 1 and Track 2 of the second DIHARD speaker diarization challenge. We describe each module in our speaker diarization pipeline and explain the rationale behind our choice of algorithms for each module, while comparing the Diarization Error Rate (DER) against different module combinations. We propose a clustering scheme based on spectral clustering that yields competitive performance. Moreover, we introduce an overlap detection scheme and a re-segmentation system for speaker diarization and investigate their performances using controlled and in-the-wild conditions. In addition, we describe the additional components that will be integrated to our speaker diarization system. To pursue the best performance, we compare our system with the state-of-the-art methods that are presented in the previous challenge and literature. We include preliminary results of our speaker diarization system on the evaluation data from the second DIHARD challenge.
BibTeX
@inproceedings{park2019second,
  author = {Tae Jin Park and Manoj Kumar and Nikolaos Flemotomos and Monisankha Pal and Raghuveer Peri and Rimita Lahiri and Panayiotis G. Georgiou and Shrikanth Narayanan},
  title = {{The Second DIHARD Challenge: System Description for USC-SAIL Team}},
  booktitle = {Interspeech},
  year = {2019},
  url = {https://openreview.net/forum?id=C5cPQJh4zv},
}
Figure