A hybrid approach to online speaker diarization

Carlos Vaquero, Oriol Vinyals, Gerald FriedlandInterspeech 2010 · DOI

Abstract

Speaker diarization—the task of determining "who spoke when"—is traditionally an offline process requiring access to the entire audio stream. This paper introduces a hybrid architecture that enables near real-time diarization with a latency of just 2.6 seconds. By leveraging a high-accuracy offline "teacher" subsystem to continuously refine a low-latency online "student" subsystem, the proposed framework identifies speakers without prior voice models or knowledge of the number of participants.

Introduction

Traditional speaker diarization systems are designed for offline processing, where global data allows for high-accuracy clustering of speaker identities. However, emerging applications like interactive robotics and live closed-captioning require online diarization—identifying the current speaker with minimal delay. The primary challenge in online settings is the scarcity of data, which often leads to poor model initialization and error propagation. This paper addresses these issues by using a background offline process to provide reliable labels for an online identification system.

System Architecture

The proposed hybrid system consists of two interacting subsystems:

  1. The Offline Subsystem (The Teacher): This component processes the entire audio history from the start of the session. It utilizes the ICSI diarization system to generate high-accuracy speaker labels.
  2. The Online Subsystem (The Student): This component processes live audio using a 2.5-second buffer. It adapts a Universal Background Model (UBM) via Maximum a Posteriori (MAP) estimation using the labels provided by the offline system.
Hybrid architecture of the online and offline subsystems.

A critical feature of this architecture is the "best speaker mapping" mechanism. Since diarization is unsupervised, the offline system must ensure that speaker IDs remain consistent across successive updates (e.g., "Speaker 1" remains "Speaker 1"). These refined models are then used by the online system to make final decisions via majority voting over short windows.

Experimental Setup

The system was evaluated using a subset of 26 meetings from the NIST Rich Transcription 2009 (Dev09) evaluation set, featuring varied environments and multiple speakers. For initial training, the AMI Meeting Corpus was used to build a gender-independent UBM with 128 Gaussians, providing a baseline understanding of speech characteristics before any specific meeting data was processed.

Key Results and Insights

The hybrid system achieves a decision latency of approximately 2.6 seconds. While the offline system achieves a Diarization Error Rate (DER) of 18.11% on full audio, the hybrid online system reaches a DER of 37.75%. This is a significant achievement for a system starting with zero prior knowledge of the speakers.

Evolution of the Diarization Error Rate (DER) over time.

Several key insights emerged from the study:

  • Warm-up Phase: The system requires approximately 60 seconds of initial audio for the offline subsystem to generate the first set of reliable speaker models.
  • Speed vs. Accuracy: Interestingly, increasing the speed of the offline "teacher" can improve the online "student." Faster processing allows the system to detect new speakers and update models more quickly, reducing the period during which the online system is "blind" to new participants.
  • Model Complexity: The researchers found that using 5 Gaussians for the offline models offered the best balance between computational efficiency and accuracy.

Limitations and Future Work

The current system is primarily limited by the "new speaker gap"—the time required for the offline subsystem to identify a new participant and update the online models. If a speaker only contributes a brief segment between updates, they may be misidentified. Future work will focus on parallelizing the offline subsystem to reduce update lag and further narrow the performance gap between offline and online diarization.

Summary updated

BibTeX
@inproceedings{vaquero2010hybrid,
  author = {Carlos Vaquero and Oriol Vinyals and Gerald Friedland},
  title = {{A hybrid approach to online speaker diarization}},
  booktitle = {Interspeech},
  year = {2010},
  doi = {10.21437/interspeech.2010-700},
}
Figure