Abstract
Speaker clustering is a critical component of speaker diarization, aimed at grouping speech segments by individual identity without prior knowledge. Traditional Agglomerative Hierarchical Clustering (AHC) often relies on either overly simplistic single Gaussian models or computationally intensive Gaussian Mixture Models (GMMs) trained via Expectation-Maximization (EM). This paper proposes an "incremental Gaussian mixture modeling" approach that constructs cluster models through a recursive weighted sum of parent distributions. This method avoids the pitfalls of EM training, such as overfitting and high computational cost. Evaluation on 15 diverse datasets demonstrates a 4.47% absolute reduction in Speaker Error Time Rate (SETR) compared to single Gaussian baselines.
Introduction
Speaker clustering involves partitioning a set of audio segments into clusters, where each cluster ideally contains speech from a single individual. The standard technique, Agglomerative Hierarchical Clustering (AHC), starts with many small segments and iteratively merges the most similar pairs based on a distance metric, often the Bayesian Information Criterion (BIC).
The choice of statistical model for these clusters is a known bottleneck. Single Gaussian models are efficient but lack the descriptive power to represent the acoustic variety of a speaker's voice in larger clusters. Conversely, standard GMMs require EM training, which is prone to overfitting—especially when data is sparse in early clustering stages—and is computationally expensive.
Methodology: Incremental GMM
The authors introduce a method to leverage the descriptive power of GMMs without the drawbacks of iterative training. The core idea is Incremental Gaussian Mixture Cluster Modeling.
When two clusters, $X$ and $Y$, are merged into a new cluster $Z$, the model for $Z$ is defined as the weighted sum of the probability density functions (PDFs) of $X$ and $Y$:
$$f(\theta_z) = \frac{M}{M+N}f(\theta_x) + \frac{N}{M+N}f(\theta_y)$$
where $M$ and $N$ are the number of data points in clusters $X$ and $Y$, respectively.
Crucially, this process is recursive. Starting from single Gaussian initializations, the model complexity (number of components) grows naturally as clustering progresses. No EM algorithm is applied to re-fit the parameters after a merge, ensuring computational efficiency and model stability.
Experimental Evaluation
The proposed method was tested on 4 hours of audio comprising 15 datasets derived from the ICSI, NIST, and ISL meeting corpora. These datasets included 4 to 9 speakers and varied significantly in gender balance and segment lengths.
The Role of EM Training
A key insight of this research is that EM training can be counter-productive in the context of speaker clustering. As shown in Figure 1, applying EM iterations can actually decrease the inter-cluster distance (discernibility) between different speakers. EM maximizes local likelihood, which may inadvertently smooth over the unique acoustic characteristics that distinguish one speaker from another.
Results
The incremental GMM approach consistently outperformed the single Gaussian baseline across all 15 data sources. The average Speaker Error Time Rate (SETR) improved by 4.47% absolute.
As illustrated in Figure 2, the proposed method not only reduces the error rate but also provides greater stability. While the single Gaussian method's performance varied wildly (e.g., Data Source 15), the incremental method mitigated these extreme errors, suggesting better robustness to diverse acoustic environments and speaker characteristics.
Conclusion
By replacing iterative EM training with a simple incremental update rule, the authors developed a speaker clustering method that is both more accurate and more efficient than traditional approaches. This technique allows GMMs to be used effectively even with limited data, providing a scalable solution for real-world speaker diarization tasks. Future work will focus on integrating this method into end-to-end systems that handle overlapping speech and automated segment boundaries.
Summary updated