I-vectors and ILP clustering adapted to cross-show speaker diarization

Grégor Dupuy, Mickael Rouvier, Sylvain Meignier, Yannick EstèveInterspeech 2012

TL;DR

This paper tackles the challenge of identifying the same speaker across multiple audio recordings (cross-show speaker diarization). They propose a novel approach using i-vectors and Integer Linear Programming (ILP) for clustering, which significantly speeds up the process compared to existing methods while maintaining comparable accuracy. This makes it more practical for analyzing large audio collections.

What Problem Does This Paper Address?

Speaker diarization, the task of segmenting an audio stream and assigning segments to specific speakers, has traditionally been performed on individual audio files (shows). A major limitation of this approach is that a speaker appearing in multiple shows would be assigned different anonymous labels in each show. This is problematic for applications dealing with broadcast news or other collections where recurring speakers (e.g., hosts, journalists) are common. The paper addresses this by focusing on "cross-show speaker diarization," aiming to consistently identify the same speaker across an entire collection of shows using a single, consistent label.

Methods

The authors build upon a two-stage architecture:

  1. Single-show diarization: Each show is processed individually using the LIUM_SpkDiarization system. This system employs acoustic segmentation, Gaussian Mixture Models (GMMs) for speaker modeling, and hierarchical agglomerative clustering using the Bayesian Information Criterion (BIC) and Normalized Cross-Likelihood Ratio (NCLR).
  2. Cross-show diarization: This stage aims to group speakers appearing in multiple shows. They compare two methods for this overall processing:
  3. NCLR-based clustering: Similar to previous hybrid approaches, this method uses NCLR for clustering across the concatenated outputs of individual shows.
  4. I-vectors and ILP clustering: This is the novel contribution. They extract i-vectors from segments identified in the single-show diarization. I-vectors are compact representations of speaker characteristics. The cross-show clustering problem is then formulated as an Integer Linear Programming (ILP) problem. The objective is to minimize the number of speaker clusters and the dispersion of i-vectors within each cluster, subject to constraints that ensure each segment is assigned to a single speaker. This ILP formulation is solved using a Branch and Bound algorithm.

Datasets

The experiments were conducted on the ESTER 2 French evaluation campaign training corpus, which comprises 100 hours of manually transcribed French radio broadcast news recorded between 1999 and 2003. This corpus was divided into thirteen smaller subsets (one development corpus and twelve test corpora) for independent experiments.

Key Insights and Ideas

  • I-vectors as a compact representation for cross-show clustering: The paper demonstrates that i-vectors, commonly used in speaker verification, can be effectively leveraged for speaker clustering in diarization, especially when dealing with the complexity of cross-show identification.
  • ILP for efficient and effective clustering: Formulating the cross-show clustering as an ILP problem provides a powerful way to optimize for both the number of speakers and the consistency of speaker assignments. This approach offers a more principled way to handle the grouping compared to purely hierarchical methods.
  • Significant speedup with ILP: The most striking insight is the dramatic reduction in computational time achieved by the ILP-based approach, making it highly practical for large-scale audio analysis.
  • Trade-off between single-show and cross-show DER: While overall cross-show DER is the primary metric, the paper also reports single-show DER. Interestingly, the ILP system showed slightly better single-show DER in some cases, suggesting its clustering doesn't negatively impact the initial segmentation quality.

Key Results

  • Accuracy: The ILP-based cross-show diarization system achieved comparable performance to the NCLR-based system in terms of both single-show and cross-show Diarization Error Rate (DER). The cross-show DER for the ILP system was 14.21% on average, versus 14.73% for the NCLR system.
  • Speed: The ILP clustering method was, on average, 8.66 times faster than the NCLR clustering method. This speedup varied significantly across corpora, ranging from 2.90 to 17.67 times faster. For example, processing 10-hour corpora took 1:58 hours with ILP versus 10:57 hours with NCLR.
  • Overall Improvement: In some cases, the cross-show clustering (both NCLR and ILP) improved the single-show DER compared to processing shows individually. The average single-show DER was 9.06% for individual processing, 8.69% with overall NCLR clustering, and 8.85% with overall ILP clustering.

Limitations and Future Work

  • Performance on corpora with high female speaker proportion: The authors noted poorer results on specific test corpora where the proportion of female speakers was twice as high. They suggest that using gender-dependent UBMs in the overall clustering stage could improve performance and reduce computational time.
  • Adding new shows: While the ILP system is faster, adding new shows still requires re-processing the entire collection. However, the authors mention that already learned speaker models can be reused, mitigating some of this overhead.
  • Further optimization: The paper suggests that gender detection during individual processing could be leveraged for more refined overall clustering.

Summary updated

BibTeX
@inproceedings{dupuy2012i,
  author = {Grégor Dupuy and Mickael Rouvier and Sylvain Meignier and Yannick Estève},
  title = {{I-vectors and ILP clustering adapted to cross-show speaker diarization}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure