TL;DR
eDIAR is a training-free method that turns fixed-slot neural diarizers (like Nemotron-3) into "open-speaker" systems capable of handling an unlimited number of participants. By combining the stability of neural slot-based models with the efficiency of the e-means clustering algorithm, it maintains high accuracy in long meetings and enables frame-level speaker linking across massive archives at a fraction of the usual computational cost.
What Problem Does This Paper Address?
Speaker diarization—the task of determining "who spoke when"—faces a significant hurdle in real-world scenarios: scale. State-of-the-art end-to-end neural diarizers typically use a fixed number of output "slots" (e.g., 4 or 8). While accurate and fast, they break down when a ninth person joins the conversation.
Conversely, traditional clustering-based pipelines can handle any number of speakers but often suffer from "identity drift" in streaming settings and become computationally prohibitive when trying to link speakers across large archives of recordings. The authors sought a way to bridge this gap: a system that retains the robustness of neural models while gaining the unbounded capacity of clustering, all without the need for expensive retraining.
Methods
The authors introduce a hybrid approach that adds a lightweight clustering layer on top of existing fixed-slot models:
- Capacity-Aware Splitting: The system trusts the fixed-slot "head" (like Nemotron-3) for activity detection and overlap handling. Clustering is only triggered for a specific slot when all available slots are already occupied, effectively "splitting" a slot to host multiple voices.
- e-means Algorithm: Instead of standard k-means, they use e-means, a fast mini-batch variant that uses exponential moving averages for updates. It reaches optimal clustering quality significantly faster than traditional methods.
- Streaming with Revision: To keep compute costs constant, re-clustering is performed within a 90-second sliding window. The system allows for "retroactive" label revisions for up to 10 seconds, correcting early mistakes as more audio context becomes available.
- Slot-Anchored Identities: Speaker identities are anchored to the neural head's slots, providing long-term stability that pure clustering systems lack.
- Large-Scale Linking: The efficiency of e-means allows the system to link speakers across entire archives (millions of frames) by clustering frame-level embeddings directly, rather than relying on less accurate session-level centroids.
Datasets
The system was evaluated on a mix of real-world and synthetic datasets:
- VoxConverse: Real YouTube clips featuring diverse speakers and environments.
- AMI & ICSI: Standard meeting corpora used to test performance in natural, multi-party conversations.
- LibriSpeech-OpenSpk: A synthetic dataset created by the authors to specifically test "stress cases" with up to 20 speakers and hour-long durations.
- VoxCeleb1: Used to benchmark the speaker-linking task across an archive of 1,248 sessions.
Key Insights and Ideas
- "Trust but Verify": The core philosophy is to trust the neural head for what it does best (activity and overlap) and only use clustering to solve the specific problem of capacity limits.
- Structural Robustness: By confining re-clustering to a bounded window and anchoring IDs to slots, the authors prevent errors from "drifting" and accumulating over the course of a long meeting.
- Frame-Level Accuracy: Traditional archive linking often clusters one average "centroid" per speaker per session. eDIAR shows that clustering every single frame is now computationally feasible with e-means and leads to much higher linking accuracy.
- Plug-and-Play: Because it requires no retraining, eDIAR can be immediately applied to improve existing deployed models like Nemotron-3 or Streaming Sortformer.
Key Results
The results, summarized in Figure 1, demonstrate significant improvements over baseline fixed-slot models:
- Breaking the Speaker Cap: In conversations with 20 speakers, eDIAR reduced the Diarization Error Rate (DER) from 66% to 44% for Nemotron-3, and from 81% to 44% for the 4-slot Streaming Sortformer.
- Long-Term Stability: On hour-long meetings, eDIAR remained stable while Streaming Sortformer's error rate collapsed, rising from 23% to 57%.
- Efficiency: The system operates well within real-time budgets. As shown in Figure 1(d), the time per update stays flat even as the meeting length increases, whereas re-clustering the entire history would eventually exceed the real-time budget.
- Archive Speedup: For large-scale linking, e-means reached the same clustering loss as Lloyd's k-means several times faster, making frame-level linking of million-point datasets practical on a standard CPU.
Limitations and Future Work
- Rapid Alternation: The authors note that in very long meetings where many speakers talk in rapid, short turns, the error rate can still grow. The 90-second window may not contain enough turns for each speaker to maintain distinct clusters.
- Archive k-Selection: For archive linking, the number of speakers ($k$) currently needs to be provided to the algorithm. Automating the estimation of $k$ for massive archives is a potential area for future research.
- Scope: The current evaluation focused on English speech; performance on other languages or in extremely challenging acoustic environments (e.g., high reverberation) remains to be fully explored.
Summary updated