Speaker diarization using divide-and-conquer

Shih-Sian Cheng, Chun-Han Tseng, Chia-Ping Chen, Hsin-Min WangInterspeech 2009 · DOI

Abstract

Speaker diarization, the process of partitioning an audio stream into homogeneous segments according to speaker identity, faces significant scalability challenges due to the quadratic computational complexity of traditional clustering methods. This paper presents a "Divide-and-Conquer" (DAC) framework designed to accelerate the diarization process. By recursively partitioning the audio into smaller sub-streams and performing localized clustering, the proposed system achieves up to a 20x speedup on large datasets with negligible impact on accuracy.

Introduction

In speech processing, speaker diarization answers the fundamental question: "Who spoke when?" It is an essential step for indexing broadcast news, transcribing meetings, and organizing large audio archives. The standard approach involves two main stages: segmentation (detecting speaker changes) and clustering (grouping segments by speaker).

The bottleneck in this pipeline is typically Hierarchical Agglomerative Clustering (HAC). HAC starts with each segment as a separate cluster and iteratively merges the most similar pairs. Because it requires comparing all clusters at each step, its computational complexity is $O(N^2)$, where $N$ is the number of segments. As audio recordings grow in length, the time required for clustering increases exponentially, making real-time or large-scale processing difficult.

The DACDiar Framework

To address the $O(N^2)$ bottleneck, the authors introduced DACDiar, a framework that applies a divide-and-conquer strategy to the diarization task. The process is split into two primary phases: identifying "divide-points" and recursively processing the resulting sub-streams.

1. Finding the Divide-Points

The system first analyzes the audio stream to find natural breaking points where speaker changes are highly likely. This is achieved using a sliding window to calculate the Generalized Likelihood Ratio (GLR). Peaks in the GLR distance curve indicate potential boundaries between different speakers.

The "Divide" process: from audio stream to detecting divide-points via GLR distance curves.

2. Recursive Splitting and Combining

Once divide-points are identified, the algorithm recursively partitions the speech stream at the most prominent peak, building a tree structure. This continues until sub-streams contain only a single segment.

The "Conquer" phase involves moving back up the tree. At each node, the system uses HAC to merge clusters from its child sub-streams. Because HAC operates on a significantly smaller number of clusters at each local step compared to the entire dataset, the overall computational load is drastically reduced.

The recursive tree structure showing how segments are grouped and merged hierarchically.

Experimental Evaluation

The researchers evaluated DACDiar using NIST benchmark datasets: RT-02 (10-minute news shows) and RT-03 (30-minute news shows). They tested three distance metrics for clustering:

  • BIC (Bayesian Information Criterion): Fast but less precise.
  • SID (Speaker Identification): High accuracy using GMM-UBM techniques but computationally intensive.
  • BIC-SID Hybrid: Uses BIC for initial coarse clustering and SID for final refinement.

Key Results

The DAC framework demonstrated remarkable efficiency gains over the baseline HAC system while maintaining comparable Diarization Error Rates (DER).

Method Dataset Speedup Baseline DER DAC DER
DACDiar_BIC RT-03 20.0x 21.01% 21.16%
DACDiar_SID RT-03 4.1x 18.04% 18.27%
DACDiar_BIC_SID RT-03 2.18x 17.15% 17.45%

The results show that the speedup is more pronounced as the audio length increases, confirming the framework's scalability. The hybrid BIC-SID approach provided the best balance, offering a significant speed boost with minimal accuracy loss.

Conclusion

The Divide-and-Conquer framework effectively bypasses the quadratic complexity of traditional speaker diarization. By localizing the clustering process, DACDiar enables the processing of long-form audio recordings that were previously computationally prohibitive. Future work may involve integrating Viterbi re-segmentation to further refine accuracy and incorporating metadata like gender or channel characteristics to improve the hierarchical merging process.

Reference: Cheng, S.-S., Tseng, C.-H., Chen, C.-P., & Wang, H.-M. (2009). Speaker Diarization Using Divide-and-Conquer. Proc. Interspeech 2009, Brighton, UK.

Summary updated

BibTeX
@inproceedings{cheng2009speaker,
  author = {Shih-Sian Cheng and Chun-Han Tseng and Chia-Ping Chen and Hsin-Min Wang},
  title = {{Speaker diarization using divide-and-conquer}},
  booktitle = {Interspeech},
  year = {2009},
  doi = {10.21437/interspeech.2009-324},
}
Figure