Speaker clustering is a fundamental component of speaker diarization—the process of determining "who spoke when" in an audio recording. This task involves partitioning an audio stream into groups based on speaker identity without prior knowledge of the number of speakers. Traditional methods often struggle with either sensitivity to initial errors or extreme over-segmentation. This paper proposes a hybrid approach, APAHC, which combines the strengths of Affinity Propagation (AP) and Agglomerative Hierarchical Clustering (AHC) to achieve more robust results across diverse audio environments.
The Clustering Dilemma: AHC vs. AP
Historically, Agglomerative Hierarchical Clustering (AHC) has been the standard. It begins with numerous small clusters and merges them based on distance metrics until a stopping criterion, such as the Bayesian Information Criterion (BIC), is reached. However, AHC is highly sensitive to early merging errors; if two different speakers are incorrectly combined in the first few steps, the error propagates throughout the process.
Affinity Propagation (AP) emerged as a powerful alternative that does not require a pre-defined number of clusters. By passing "responsibility" and "availability" messages between data points, it identifies exemplars that represent cluster centers. While AP is excellent at minimizing local error, it tends to "over-cluster," splitting a single speaker's voice into too many groups. This results in high cluster purity (each group contains only one person) but poor speaker purity (one person's voice is scattered across many groups).
The Proposed APAHC Framework
The authors introduce APAHC (Affinity Propagation + Agglomerative Hierarchical Clustering), a two-stage pipeline designed to leverage the local precision of AP and the global merging capabilities of AHC:
- Stage 1: Affinity Propagation (AP): Every speech segment is treated as a node in a network. Similarities are calculated using the Negative Generalized Likelihood Ratio (GLR) between Gaussian models of the segments. AP is tuned to "under-cluster," creating many small, highly pure clusters that serve as clean building blocks.
- Stage 2: Agglomerative Merging (AHC): These pure initial clusters are then processed via AHC. The system trains a new Gaussian model for each cluster and uses the $\Delta BIC$ as a stopping criterion to merge clusters until only the distinct speakers remain.
By using AP as a sophisticated pre-processor, the AHC stage avoids the "poisoning" of clusters that occurs when raw, individual speech segments are merged incorrectly at the start.
Experimental Results
The researchers evaluated APAHC against standalone AHC and AP using two distinct datasets: Telephone Conversations (NIST 2004) and Broadcast News (MDE RT-04). Performance was measured using cluster purity ($cp$), speaker purity ($sp$), and an overall criterion $K$ (the geometric mean of $cp$ and $sp$).
Telephone Data Performance
On the NIST 2004 dataset, which consists of 48 conversations of roughly 5 minutes each, APAHC demonstrated superior stability and overall accuracy.
As shown in the table above, APAHC achieved a $K$ score of 92.9%, outperforming both standalone AHC (90.7%) and AP (89.3%). Notably, it significantly improved speaker purity ($sp$) compared to the other methods.
Broadcast News Performance
The MDE RT-04 dataset presented a greater challenge, featuring longer durations and up to 40 speakers per session. In this environment, the over-clustering issue of standalone AP became most apparent.
In the broadcast news evaluation, standalone AP suffered from a dismal speaker purity of 53.6%. The hybrid APAHC method successfully corrected this, raising speaker purity to 83.5% and achieving the highest overall $K$ score of 88.5%.
Conclusion and Future Directions
The APAHC method proves that the synergy between AP and AHC compensates for their individual weaknesses. AP provides stable, high-purity initial centers, while AHC provides an effective global stopping mechanism. Future work may involve moving from simple Gaussian models to Gaussian Mixture Models (GMMs) and exploring more sophisticated distance metrics like the Cross Likelihood Ratio (CLR) to further refine the clustering process.
Summary updated