eDIAR: Training-Free Open-Speaker Streaming Diarization for Long Meetings and Large Archives via e-means

Kyudan Jung, Jihwan Kim, Minwoo Lee, Eunji Shin, Jaegul ChooarXiv 2026 · arXiv

TL;DR

eDIAR is a training-free method that turns fixed-slot neural diarizers (like Nemotron-3) into "open-speaker" systems capable of handling an unlimited number of participants. By combining the stability of neural slot-based models with the efficiency of the e-means clustering algorithm, it maintains high accuracy in long meetings and enables frame-level speaker linking across massive archives at a fraction of the usual computational cost.

What Problem Does This Paper Address?

Speaker diarization—the task of determining "who spoke when"—faces a significant hurdle in real-world scenarios: scale. State-of-the-art end-to-end neural diarizers typically use a fixed number of output "slots" (e.g., 4 or 8). While accurate and fast, they break down when a ninth person joins the conversation.

Conversely, traditional clustering-based pipelines can handle any number of speakers but often suffer from "identity drift" in streaming settings and become computationally prohibitive when trying to link speakers across large archives of recordings. The authors sought a way to bridge this gap: a system that retains the robustness of neural models while gaining the unbounded capacity of clustering, all without the need for expensive retraining.

Methods

The authors introduce a hybrid approach that adds a lightweight clustering layer on top of existing fixed-slot models:

  1. Capacity-Aware Splitting: The system trusts the fixed-slot "head" (like Nemotron-3) for activity detection and overlap handling. Clustering is only triggered for a specific slot when all available slots are already occupied, effectively "splitting" a slot to host multiple voices.
  2. e-means Algorithm: Instead of standard k-means, they use e-means, a fast mini-batch variant that uses exponential moving averages for updates. It reaches optimal clustering quality significantly faster than traditional methods.
  3. Streaming with Revision: To keep compute costs constant, re-clustering is performed within a 90-second sliding window. The system allows for "retroactive" label revisions for up to 10 seconds, correcting early mistakes as more audio context becomes available.
  4. Slot-Anchored Identities: Speaker identities are anchored to the neural head's slots, providing long-term stability that pure clustering systems lack.
  5. Large-Scale Linking: The efficiency of e-means allows the system to link speakers across entire archives (millions of frames) by clustering frame-level embeddings directly, rather than relying on less accurate session-level centroids.

Datasets

The system was evaluated on a mix of real-world and synthetic datasets:

  • VoxConverse: Real YouTube clips featuring diverse speakers and environments.
  • AMI & ICSI: Standard meeting corpora used to test performance in natural, multi-party conversations.
  • LibriSpeech-OpenSpk: A synthetic dataset created by the authors to specifically test "stress cases" with up to 20 speakers and hour-long durations.
  • VoxCeleb1: Used to benchmark the speaker-linking task across an archive of 1,248 sessions.

Key Insights and Ideas

  • "Trust but Verify": The core philosophy is to trust the neural head for what it does best (activity and overlap) and only use clustering to solve the specific problem of capacity limits.
  • Structural Robustness: By confining re-clustering to a bounded window and anchoring IDs to slots, the authors prevent errors from "drifting" and accumulating over the course of a long meeting.
  • Frame-Level Accuracy: Traditional archive linking often clusters one average "centroid" per speaker per session. eDIAR shows that clustering every single frame is now computationally feasible with e-means and leads to much higher linking accuracy.
  • Plug-and-Play: Because it requires no retraining, eDIAR can be immediately applied to improve existing deployed models like Nemotron-3 or Streaming Sortformer.

Key Results

The results, summarized in Figure 1, demonstrate significant improvements over baseline fixed-slot models:

Figure 1: eDIAR at a glance. Streaming at 1.04 s. (a) DER against the number of speakers, from two to twenty, on LibriSpeech-OpenSpk conversations, compared with the two fixed-slot models; the dotted lines mark four and eight slots. (b) DER of six one-hour meetings with 12 to 20 speakers (LibriSpeech-OpenSpk utterances, speakers joining over time), scored on their first 15, 30 and 45 minutes. (c) Linking the speakers of 1,248 sessions with 1,251 VoxCeleb1 speakers across recordings: archive DER against clustering time at the level of 1.16 million frames; numbers next to the markers are passes over the data. (d) Processing time per update over the same long recordings, for our bounded 90 s window and for re-clustering the whole history, with the real-time budget of one 0.72 s chunk.
  • Breaking the Speaker Cap: In conversations with 20 speakers, eDIAR reduced the Diarization Error Rate (DER) from 66% to 44% for Nemotron-3, and from 81% to 44% for the 4-slot Streaming Sortformer.
  • Long-Term Stability: On hour-long meetings, eDIAR remained stable while Streaming Sortformer's error rate collapsed, rising from 23% to 57%.
  • Efficiency: The system operates well within real-time budgets. As shown in Figure 1(d), the time per update stays flat even as the meeting length increases, whereas re-clustering the entire history would eventually exceed the real-time budget.
  • Archive Speedup: For large-scale linking, e-means reached the same clustering loss as Lloyd's k-means several times faster, making frame-level linking of million-point datasets practical on a standard CPU.

Limitations and Future Work

  • Rapid Alternation: The authors note that in very long meetings where many speakers talk in rapid, short turns, the error rate can still grow. The 90-second window may not contain enough turns for each speaker to maintain distinct clusters.
  • Archive k-Selection: For archive linking, the number of speakers ($k$) currently needs to be provided to the algorithm. Automating the estimation of $k$ for massive archives is a potential area for future research.
  • Scope: The current evaluation focused on English speech; performance on other languages or in extremely challenging acoustic environments (e.g., high reverberation) remains to be fully explored.

Summary updated

Abstract

Meetings last up to an hour, and nobody knows in advance how many people will speak. Neural diarizers with a fixed number of output slots, such as Streaming Sortformer and Nemotron-3, are accurate and run online, but they cannot name more speakers than they have slots. We remove this limit without retraining. The slot head keeps deciding who is active and when speech overlaps, and speaker embeddings are clustered inside a single slot, with the official e-means algorithm, only after every slot has been used. In the streaming system, re-clustering is confined to a bounded window, labels are revised for a few seconds, and identities are anchored to the slots of the head. The resulting system has no limit on the number of speakers and stays accurate when speakers outnumber the slots, where fixed-slot models break down. On meetings that grow to an hour it stays far more accurate than Streaming Sortformer, which accumulates errors, and its cost per update does not grow with the meeting. The same recipe turns the four-slot Streaming Sortformer into an open-speaker system. Across recordings, e-means makes it affordable to link the speakers of a large archive frame by frame, a task that arises when conversational corpora are prepared, and it reaches the clustering quality of Lloyd's $k$-means several times faster. The method needs little tuning, and e-means runs with its default recipe from a few hundred to more than a million embeddings.
BibTeX
@misc{jung2026ediar,
  author = {Kyudan Jung and Jihwan Kim and Minwoo Lee and Eunji Shin and Jaegul Choo},
  title = {{eDIAR: Training-Free Open-Speaker Streaming Diarization for Long Meetings and Large Archives via e-means}},
  year = {2026},
  eprint = {2611.18529},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18529},
}
Figure