Scaling Speaker Diarization to Large Audio Corpora

45 concepts42 papers

The Monolithic Bottleneck

When speaker diarization algorithms are applied to massive collections like radio archives or multi-session meetings, they hit a severe computational wall. The core issue lies in the mathematical complexity of clustering. Standard diarization systems often rely on iterative statistical models like Hidden Markov Models and Gaussian Mixture Models (HMM/GMM) or perform exhaustive pairwise comparisons between speech segments. If a dataset grows from minutes to thousands of hours, the number of segments increases linearly, but the cost of pairwise matching or global optimization scales quadratically.

As highlighted by Man-hung Siu et al. in Unsupervised audio patterns discovery using HMM-based self-organized units, exhaustive pairwise comparisons make it nearly impossible to scale pattern discovery across multiple documents simultaneously. If we attempt a naive "monolithic" approach—treating an entire collection of audio shows as a single continuous stream—the memory and processing requirements become completely prohibitive. Viet-Anh Tran et al. demonstrate in Comparing multi-stage approaches for cross-show speaker diarization that this monolithic strategy demands excessive resources, rendering it impractical for extensive archives. Similarly, Hangting Chen et al. note in AutoPrep: An Automatic Preprocessing Framework for In-the-Wild Speech Data that high computational complexity typically restricts clustering algorithms to short segments under two hours, leaving a significant gap in our ability to process continuous, long-form audio streams.

Local-to-Global Architectures

To bypass this bottleneck, researchers use a hierarchical "divide-and-conquer" paradigm. Instead of performing a single massive clustering operation over millions of audio frames, the problem is split into two distinct, localized phases.

Shih-Sian Cheng et al. formalized this framework in Speaker diarization using divide-and-conquer. The intuition is straightforward: first, perform local diarization on individual files or short audio segments. Because each file contains only a few speakers and a limited duration, the local clustering problem is small, fast, and highly parallelizable. Second, perform global speaker linking. Instead of clustering the original raw audio segments, the global step clusters the speaker models or embeddings generated by the local step. This reduces the number of items to cluster by orders of magnitude.

For instance, Emre Yilmaz et al. applied this to a 3,000-hour radio archive in Large-Scale Speaker Diarization of Radio Broadcast Archives by extracting deep neural x-vector embeddings from local tape-level clusters and linking them globally. This bridges the gap between local precision and global tracking without pooling all raw data simultaneously. Similarly, Marc Ferràs et al. show in Diarizing large corpora using multi-modal speaker linking that linking local speaker identities across multi-session recordings using multi-modal cues drastically cuts down computational complexity while maintaining high accuracy.

Incremental and Stream-Based Processing

While a two-stage local-to-global approach works well for static archives, real-world databases grow continuously. Re-running global clustering every time a new audio file is added would recreate the same scaling bottleneck. This has driven the development of incremental processing frameworks.

Instead of pooling all segments for batch analysis, incremental systems update speaker models dynamically over time. In Investigation of Cross-Show Speaker Diarization, Qin Jin et al. compared global batch methods against incremental approaches. They found that while global methods provide high accuracy on static data, an incremental approach using T-Norm normalization offers the scalability required for growing databases. This allows the system to adapt to new audio shows without incurring the massive memory footprints of batch-oriented alternatives, as also validated by Viet-Anh Tran et al. in Comparing multi-stage approaches for cross-show speaker diarization.

Coarse Filtering and Efficient Embedding Spaces

Another critical mechanism for reducing computational overhead is avoiding fine-grained, high-fidelity comparisons between unrelated speech segments. By mapping audio into structured embedding spaces or using coarse pre-grouping techniques, systems can filter out irrelevant data early in the pipeline.

Feng Chen et al. address this in A Robust Speaker Clustering Method Based on Discrete Tied Variational Autoencoder. By utilizing a Discrete Tied Variational Autoencoder (DTVAE) with discrete latent variables, their system pre-groups speakers into coarse clusters based on mutual information. This initial partitioning avoids the exhaustive comparison of new entries against every single record in a massive database, slashing processing time by up to 32%.

A similar philosophy applies to broader audio retrieval and pattern discovery tasks. For example, Deepu Vijayasenan et al. in Integration of TDOA features in information bottleneck framework for fast speaker diarization replaced slow HMM/GMM iterations with an Information Bottleneck framework, achieving fast, real-time diarization performance. In the domain of unsupervised acoustic modeling, Haipeng Wang et al. developed Gaussian Component Clustering in A graph-based Gaussian component clustering approach to unsupervised acoustic modeling, which clusters a fixed set of Gaussian components rather than a linear explosion of individual speech segments.

By shifting from exhaustive, monolithic processing to multi-stage architectures, incremental updates, and coarse-to-fine embedding spaces, modern speech systems can scale efficiently to handle the massive volume of real-world audio data.

Key Foundational Work

Invited Talk: Processing Broadcast Audio for Information Access (2001) — contributed 1 concept elements to this direction.
A Fast Data Structure for Disk-Based Audio Editing (2002) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_computational_cost_of_speaker_diarizatio_124697ef,
  title = {computational cost of speaker diarization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/computational-cost-of-speaker-diarization/rdr_124697ef84474eefab9b462c378e91c4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_computational_cost_of_speaker_diarizatio_124697ef_v1,
  title = {computational cost of speaker diarization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/computational-cost-of-speaker-diarization/rdr_124697ef84474eefab9b462c378e91c4/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}

See something wrong? Report a mistake