On the Use of Spectral and Iterative Methods for Speaker Diarization

Stephen Shum, Najim Dehak, Jim Glass

Abstract

Speaker diarization, the process of determining "who spoke when" in an audio recording, faces significant challenges when the number of participants is unknown. This paper extends i-vector-based diarization techniques to handle multi-party conversations. We evaluate a multi-stage pipeline combining spectral clustering for speaker count estimation and K-means clustering with cosine distance for segment grouping. Results on the NIST SRE CallHome dataset demonstrate that while spectral methods excel at estimating the number of speakers, K-means remains superior for clustering features on the unit hypersphere.

Introduction

Speaker diarization is essential for indexing and transcribing multi-speaker audio. Previous work successfully applied i-vectors—compact, low-dimensional representations of speaker characteristics—to two-party telephone calls. However, real-world scenarios often involve an unknown number of participants. This study addresses two primary hurdles: automatically estimating the speaker count and refining segment boundaries to capture rapid speaker transitions.

Methodology

The proposed system follows a multi-stage pipeline designed to move from crude audio segments to precise speaker boundaries:

  1. Feature Extraction: i-vectors are extracted to represent short audio segments. These vectors capture speaker identity while suppressing channel variability.
  2. Speaker Estimation: The system constructs an affinity matrix of segment similarities. Spectral clustering analyzes the eigenvalue decay of this matrix to heuristically estimate the number of speakers ($\hat{K}$).
  3. Clustering: With $\hat{K}$ determined, K-means clustering groups the i-vectors. Crucially, cosine distance is employed instead of Euclidean distance to account for the hyperspherical distribution of normalized i-vectors.
  4. Iterative Refinement: Initial clusters initialize Gaussian Mixture Models (GMMs) for each speaker. Viterbi alignment then refines boundaries at the acoustic frame level in an iterative process, assuming that better clusters lead to better boundaries and vice versa.

Experimental Results

The system was evaluated using the 2000 NIST SRE subset of multilingual CallHome data, comprising 500 recordings with two to seven participants. Results were compared against the state-of-the-art Castaldo 2008 system.

Clustering Performance

A key finding is the "Hypersphere Advantage." When the number of speakers is known a priori, K-means with cosine distance consistently outperforms spectral clustering across all speaker counts. This suggests that the manifold unrolling provided by spectral methods may be unnecessary or even detrimental for data already residing on a unit hypersphere.

Diarization Error Rate (DER) when the number of speakers is known a priori.

Speaker Count Estimation

While spectral clustering was less effective for the final grouping, its mathematical properties proved excellent for counting speakers. The hybrid approach—using spectral methods for estimation and K-means for clustering—achieved performance levels comparable to state-of-the-art benchmarks.

Diarization Error Rate (DER) when the number of speakers is estimated.

Discussion and Conclusion

The iterative optimization step significantly improves the Diarization Error Rate (DER) for recordings with high-quality initial hypotheses. However, the system remains sensitive to initialization and lacks temporal awareness, occasionally missing very brief speaker turns. Future research will explore probabilistic models specifically designed for hyperspherical data, such as the von Mises-Fisher distribution, and incorporate temporal dynamics via Hidden Markov Models (HMMs) to better model turn-taking behavior.

Summary updated

BibTeX
@inproceedings{shum2012use,
  author = {Stephen Shum and Najim Dehak and Jim Glass},
  title = {{On the Use of Spectral and Iterative Methods for Speaker Diarization}},
  booktitle = {Interspeech},
  year = {2012},
  doi = {10.21437/interspeech.2012-163},
}
Figure