Toeplitz Inverse Covariance Based Robust Speaker Clustering for Naturalistic Audio Streams

Harishchandra Dubey, Abhijeet Sangwan, John H.l. HansenInterspeech 2019

TL;DR

This paper introduces a novel speaker clustering method for naturalistic audio streams by modeling the temporal correlations between speaker segments using a Toeplitz Inverse Covariance (TIC) matrix within a Markov Random Field (MRF) framework. This approach significantly outperforms existing methods like cosine K-means and movMF, achieving substantial reductions in Diarization Error Rate (DER) on challenging datasets.

What Problem Does This Paper Address?

Speaker diarization, the task of identifying "who spoke when" in an audio stream, is crucial for applications like information retrieval, meeting analysis, and educational tools. However, naturalistic audio streams present significant challenges: short conversational turns, overlapping speech, noise, and reverberation. Existing speaker clustering methods often struggle with these complexities, particularly in capturing the sequential nature of how a single speaker's turns are related over time. This paper aims to address this gap by developing a clustering technique that explicitly models these temporal dependencies between speaker segments.

Methods

The core innovation of this paper lies in its use of a Toeplitz Inverse Covariance (TIC) matrix to represent the correlation network of i-vectors belonging to a single speaker. This is framed within a Markov Random Field (MRF) model, where each speaker is characterized by their own MRF. The system leverages a modified Expectation-Maximization (EM) algorithm for training.

The EM algorithm alternates between two steps:

  • E-step (Assignment): This step assigns each i-vector (a compact representation of a speech segment) to a speaker cluster. It's framed as a Viterbi path search, finding the optimal assignment sequence that balances maximizing the likelihood of the data belonging to a cluster with minimizing the number of speaker changes (controlled by a regularization parameter $\beta$).
  • M-step (Update): This step updates the model parameters for each speaker cluster. The key here is estimating the inverse covariance matrix for each cluster using a Toeplitz Graphical Lasso. This technique encourages sparsity in the inverse covariance matrix, which corresponds to conditional independence relationships within the speaker's i-vectors, and enforces a block Toeplitz structure to capture temporal dependencies. This is solved efficiently using the Alternating Direction Method of Multipliers (ADMM).

The overall pipeline involves:

  1. Ground-truth segmentation (for evaluation purposes).
  2. i-vector extraction from speech segments.
  3. Post-processing of i-vectors (mean subtraction, PCA, length normalization).
  4. The proposed TIC-based speaker clustering.

Baseline methods used for comparison include cosine K-means and Mixtures of von Mises-Fisher Distributions (movMF).

Datasets

The paper evaluates its proposed method on two datasets:

  • CRSS-PLTL corpus: This dataset consists of 80-minute audio recordings from Peer-Led Team Learning (PLTL) sessions, featuring 8 speakers. It's characterized by short conversational turns and naturalistic noise.
  • AMI Meeting corpus: Two meetings (IS1000a and IS1003b) from the AMI corpus were used, each featuring 4 speakers discussing a product design.

Key Insights and Ideas

  • Modeling Temporal Correlation is Key: The paper strongly suggests that explicitly modeling the sequential correlation between i-vectors of the same speaker is crucial for robust diarization in naturalistic audio.
  • TIC as a Powerful Representation: The Toeplitz Inverse Covariance matrix proves effective in capturing the structured temporal dependencies within a speaker's segments, going beyond simple centroid-based clustering.
  • MRF for Speaker Modeling: Framing speaker models as MRFs allows for a more nuanced representation of inter-segment relationships.
  • PCA Can Be Detrimental: Counter-intuitively, applying Principal Component Analysis (PCA) for de-correlation before the proposed TIC clustering actually degraded performance on the PLTL data. This suggests that the TIC method benefits from the inherent correlations that PCA aims to remove.
  • Robustness to Naturalistic Audio: The proposed method demonstrates significant improvements on challenging, real-world audio data, highlighting its practical applicability.

Key Results

The proposed TIC-based speaker clustering achieved significant reductions in Diarization Error Rate (DER) compared to baseline methods:

  • CRSS-PLTL Corpus: A relative DER reduction of 43.22% compared to the baseline methods.
  • AMI Meetings:
    • IS1000a: 29.37% relative DER reduction.
  • IS1003b: 9.21% relative DER reduction.

For the AMI meetings, the proposed method achieved less than 1% DER in the best-case scenarios. The paper notes that the CRSS-PLTL data is more challenging due to higher levels of distortion and more speakers, leading to generally higher DERs compared to AMI.

Limitations and Future Work

  • Ground-Truth Segmentation: The experiments were conducted using ground-truth segmentation, meaning the performance of the speech activity detection (SAD) and speaker segmentation modules were not evaluated. Future work could integrate the proposed clustering with robust SAD and segmentation techniques.
  • PCA Impact: The paper observed that PCA degraded performance for their method on PLTL data. Further investigation into how different dimensionality reduction techniques interact with TIC-based clustering could be beneficial.
  • Scalability: While the paper mentions the approach is scalable, further analysis of computational complexity for very large numbers of speakers or very long audio streams might be warranted.
  • Model Initialization: The paper mentions model initialization but doesn't detail specific strategies. Exploring different initialization techniques could potentially improve convergence speed and final performance.

Summary updated

BibTeX
@inproceedings{dubey2019toeplitz,
  author = {Harishchandra Dubey and Abhijeet Sangwan and John H.L. Hansen},
  title = {{Toeplitz Inverse Covariance Based Robust Speaker Clustering for Naturalistic Audio Streams}},
  booktitle = {Interspeech},
  year = {2019},
}

Loading related content...

Figure