Abstract: This technical report describes the EML submission to the first VoxCeleb speaker diarization challenge. Although the aim of the challenge has been the offline processing of the signals, the submitted system is basically the EML online algorithm which decides about the speaker labels in runtime approximately every 1.2 sec. For the first phase of the challenge, only VoxCeleb2 dev dataset was used for training. The results on the provided VoxConverse dev set show much better accuracy in terms of both DER and JER compared to the offline baseline provided in the challenge. The real-time factor of the whole diarization process is about 0.01 using a single CPU machine.
Introduction
Speaker diarization—the task of determining "who spoke when"—is a critical component of many speech processing pipelines. While most state-of-the-art systems operate in an "offline" mode, analyzing an entire recording at once, real-world applications often require "online" processing. Online diarization is significantly more challenging as the system must make decisions with minimal latency and without access to future context. This paper presents the EML system, an online diarization algorithm designed for "in the wild" conditions, characterized by rapid speaker exchanges, background noise, and overlapping speech.
System Architecture
The EML system is designed for high speed and discriminative power, processing audio in 0.2-second increments with a decision window of approximately 1.2 seconds.
The architecture consists of three primary components:
- Voice Activity Detection (VAD): An unsupervised GMM-based approach. It extracts zero-order Baum-Welch statistics (VAD vectors) every 20 frames and employs K-means clustering to differentiate speech from non-speech.
- Speaker Embeddings: A feedforward neural network maps audio features into a speaker-discriminative space. A key innovation is the use of the VReLU activation function, which allows the multi-layer network to be collapsed into a single transformation matrix during inference, significantly reducing computational overhead.
- Online Diarization Logic: The system uses cosine similarity to compare current speech segments against existing speaker models. To maintain reliability and prevent over-counting speakers, it employs a "split-and-compare" logic: a new speaker model is only created if two halves of a segment are similar to each other but distinct from all known models.
Experimental Setup
The system was trained primarily on the VoxCeleb2 dev set, comprising approximately 2,300 hours of audio from 6,000 speakers. For VAD training, the team supplemented this with 1,500 hours of in-house broadcast news data. Evaluation was conducted on the VoxConverse dev set, which includes 216 multi-speaker videos (roughly 20 hours) featuring heavy overlapping speech and diverse acoustic environments.
Results
The EML system demonstrated remarkable efficiency, achieving a real-time factor (RTF) of 0.01 on a single CPU. Despite its online nature, it significantly outperformed the challenge's offline baseline.
As shown in the results, the EML system achieved a Diarization Error Rate (DER) of 13.10%, nearly halving the error of the standard baseline (23.8%). The improvement was particularly pronounced in reducing "Missed Speech" (MS) and "Speaker Confusion" (SC).
Conclusion
The EML system proves that a well-optimized online diarization algorithm can surpass standard offline baselines in challenging environments. While the system performed exceptionally well on the development set, future work will focus on improving robustness to overlapping speech and refining threshold parameters to avoid overfitting to specific datasets. The use of the VReLU transformation highlights a path toward extremely low-latency, large-scale diarization applications.
Summary updated