Discriminative training for hierarchical clustering in speaker diarization

Oriol Vinyals, Gerald Friedland, Nelson MorganInterspeech 2010 · DOI

Introduction

Speaker diarization is the process of partitioning an audio stream into segments according to speaker identity—essentially answering the question "who spoke when." Traditionally, this is a "blind" problem where neither the number of speakers nor their acoustic profiles are known beforehand. Standard systems rely on Agglomerative Hierarchical Clustering (AHC) using generative models, typically Gaussian Mixture Models (GMMs) trained via Maximum Likelihood (ML).

However, generative models focus on describing the statistical distribution of a speaker's voice, which may lead to confusion when speakers share similar acoustic traits, such as pitch or room resonance. This paper, presented at INTERSPEECH 2010 by researchers from UC Berkeley and ICSI, proposes a shift from generative to discriminative modeling. By employing Maximum Mutual Information (MMI) training, the system focuses on the features that distinguish one speaker from another, rather than those that merely describe them.

Methodology

The researchers introduced a discriminative approach integrated into an unsupervised bootstrapping loop. Since true speaker labels are unavailable in diarization, the system uses the output of a standard generative AHC system as "pseudo-labels."

The Bootstrapping Loop

The process involves an iterative cycle of re-training and re-alignment. After an initial segmentation, models are retrained discriminatively. These refined models are then used to re-align the audio, providing better labels for the next iteration of clustering.

Iterative bootstrapping process for discriminative speaker diarization.

Discriminative Objective and Optimization

Instead of maximizing the probability of the data given a model, MMI maximizes the probability of the speaker identity given the data. This forces the models to ignore common acoustic features (like room acoustics) and emphasize distinctive ones. To maintain mathematical consistency during clustering, the authors derived a modified Bayesian Information Criterion (BIC) formula. Optimization was performed using an Extended Baum-Welch algorithm to ensure stable parameter updates and positive variances.

Experimental Setup

The method was evaluated on NIST Rich Transcription (RT) datasets, which are the benchmark for meeting diarization:

  • Dev07: A development set comprising 21 meetings from previous NIST evaluations.
  • Eval07 & Eval09: Official NIST evaluation sets used to test final performance.

Results and Discussion

The MMI-based approach yielded significant improvements in Diarization Error Rate (DER) across all tested datasets. On the Dev07 set, the error dropped from a baseline of 17.58% to 13.72%. Similar gains were observed on the evaluation sets, with Eval07 improving from 19.11% to 15.8% and Eval09 from 29.13% to 24.31%.

Diarization Error Rate (DER) as a function of the f parameter, comparing baseline and MMI methods.

A key finding was the robustness of the MMI approach. Even when initial pseudo-labels were of poor quality (DER > 25%), discriminative training still provided a performance boost, although the gains were most pronounced when the initial labels were relatively accurate. The system also utilized a heuristic filtering technique, only applying updates when local model decisions matched global Viterbi alignments, preventing the learning of noisy or uncertain segments.

Conclusion

The transition to discriminative training represents a significant advancement in unsupervised speaker diarization. By prioritizing speaker separation over description, the authors achieved a roughly 4% absolute improvement in DER. While the method increases computational complexity—running approximately three times slower than the baseline—the accuracy gains are substantial. Future work will address challenges such as overlapped speech and the manual tuning of hyperparameters like the interpolation factor $f$.

Summary updated

BibTeX
@inproceedings{vinyals2010discriminative,
  author = {Oriol Vinyals and Gerald Friedland and Nelson Morgan},
  title = {{Discriminative training for hierarchical clustering in speaker diarization}},
  booktitle = {Interspeech},
  year = {2010},
  doi = {10.21437/interspeech.2010-637},
}
Figure