Abstract
Speaker diarization—the task of determining "who spoke when"—is traditionally an offline process requiring access to the entire audio stream. This paper introduces a hybrid architecture that enables near real-time diarization with a latency of just 2.6 seconds. By leveraging a high-accuracy offline "teacher" subsystem to continuously refine a low-latency online "student" subsystem, the proposed framework identifies speakers without prior voice models or knowledge of the number of participants.
Introduction
Traditional speaker diarization systems are designed for offline processing, where global data allows for high-accuracy clustering of speaker identities. However, emerging applications like interactive robotics and live closed-captioning require online diarization—identifying the current speaker with minimal delay. The primary challenge in online settings is the scarcity of data, which often leads to poor model initialization and error propagation. This paper addresses these issues by using a background offline process to provide reliable labels for an online identification system.
System Architecture
The proposed hybrid system consists of two interacting subsystems:
- The Offline Subsystem (The Teacher): This component processes the entire audio history from the start of the session. It utilizes the ICSI diarization system to generate high-accuracy speaker labels.
- The Online Subsystem (The Student): This component processes live audio using a 2.5-second buffer. It adapts a Universal Background Model (UBM) via Maximum a Posteriori (MAP) estimation using the labels provided by the offline system.
A critical feature of this architecture is the "best speaker mapping" mechanism. Since diarization is unsupervised, the offline system must ensure that speaker IDs remain consistent across successive updates (e.g., "Speaker 1" remains "Speaker 1"). These refined models are then used by the online system to make final decisions via majority voting over short windows.
Experimental Setup
The system was evaluated using a subset of 26 meetings from the NIST Rich Transcription 2009 (Dev09) evaluation set, featuring varied environments and multiple speakers. For initial training, the AMI Meeting Corpus was used to build a gender-independent UBM with 128 Gaussians, providing a baseline understanding of speech characteristics before any specific meeting data was processed.
Key Results and Insights
The hybrid system achieves a decision latency of approximately 2.6 seconds. While the offline system achieves a Diarization Error Rate (DER) of 18.11% on full audio, the hybrid online system reaches a DER of 37.75%. This is a significant achievement for a system starting with zero prior knowledge of the speakers.
Several key insights emerged from the study:
- Warm-up Phase: The system requires approximately 60 seconds of initial audio for the offline subsystem to generate the first set of reliable speaker models.
- Speed vs. Accuracy: Interestingly, increasing the speed of the offline "teacher" can improve the online "student." Faster processing allows the system to detect new speakers and update models more quickly, reducing the period during which the online system is "blind" to new participants.
- Model Complexity: The researchers found that using 5 Gaussians for the offline models offered the best balance between computational efficiency and accuracy.
Limitations and Future Work
The current system is primarily limited by the "new speaker gap"—the time required for the offline subsystem to identify a new participant and update the online models. If a speaker only contributes a brief segment between updates, they may be misidentified. Future work will focus on parallelizing the offline subsystem to reduce update lag and further narrow the performance gap between offline and online diarization.
Summary updated