Speaker diarization is the task of labeling who spoke when in a recording. Streaming diarization systems keep a small memory of the speakers heard so far. They have to make two different judgments about each remembered speaker. The first is identity: what does this voice sound like? The second is presence: is this person still in the conversation and likely to talk again soon? Current systems merge the two. Either every speaker stays active forever, or speakers are dropped when a fixed-size buffer overflows. The cost shows up in long, many-party sessions. Systems built to handle an unknown number of speakers degrade once more than four or five people take part, and a person who returns after ten minutes of silence is easily mistaken for a newcomer or for someone else.
We propose to treat each speaker's talk as a sparse, irregularly timed stream of events. This borrows from work on irregularly sampled time series, where a model's trust in an old observation decays with the elapsed time since it was made. We make one change that matters for speech. The voice identity of a remembered speaker never decays. What decays is a learned estimate of whether that speaker is still present, and its rate of decay depends on elapsed silence measured in seconds. The active memory then holds only speakers who are probably present. Absent speakers move to a dormant archive and can be matched again when they return. As a result, the total number of speakers a session can have is no longer tied to the size of the network's output.
Core Claim
A streaming diarizer gets two benefits from keeping a fixed-size active speaker set governed by a learned, elapsed-time survival model of speaker presence, with identity vectors stored separately and held fixed. First, its accuracy should stay roughly constant as the number of speakers in a session grows past the four-to-six range where current end-to-end systems break down. Second, it should re-identify speakers who return after long silences more reliably than recency eviction, fixed timeouts, or simply enlarging the cache. A secondary claim follows from measuring time in seconds rather than frames. When audio packets are dropped, the decay continues through the missing span instead of pausing, so the memory treats a dropout as elapsed time rather than as unchanged state.
The idea comes from a specific tension in prior work. Temporal Belief Memory gates an RNN with a belief that decays exponentially with time since a clinical variable was last measured, eventually falling back to the population mean. It reported AUC 0.7840 for septic shock prediction, against 0.7078 for mean substitution. Applied directly to speaker embeddings, that rule would be wrong: a voice does not drift toward an average voice during silence. Only the chance that the person speaks again changes. Our design keeps the decay mechanism from that line of work but applies it to presence and not to identity. We include identity decay as an ablation, and we expect it to hurt.
Relation to Prior Work
Prior speaker-count methods fall into three groups, and each is a near miss for this proposal. The first group generates speakers dynamically but keeps them fixed for the whole session. EEND-EDA produces a variable number of global speaker vectors ("attractors") from a recurrent encoder-decoder, with one existence probability per speaker. AED-EEND replaces that decoder with attention and uses short enrollment clips. Both are offline, and both treat existence as a single per-recording decision with no notion of when the speaker was last heard. The second group over-provisions a fixed number of output slots. TS-VAD with improved i-vector estimation fixes N=8, fills unused slots with dummy vectors, and depends on a fused clustering initialization. Separating varying numbers of sources with auxiliary autoencoding trains surplus output channels to reproduce the input mixture, which makes them easy to detect and discard. In both, capacity is still a fixed architectural constant. The third group orders speakers by arrival. Streaming Sortformer keeps an arrival-order speaker cache that resolves label ambiguity across chunks without explicit matching, and Serialized Output Training orders transcripts first-in-first-out by start time. Sortformer is the closest existing system and the one we build on. Its cache is a frame buffer with a hard four-speaker bound, and nothing in it represents how long a speaker has been silent.
On the time-series side, continuous-time autoencoders and continuous-discrete state-space models handle arbitrary observation gaps but track a single latent trajectory of fixed size. MTM handles channels that are sampled at different times, but its set of channels is fixed in advance. We take only the core idea from this line: trust should be a function of elapsed physical time. We deliberately do not adopt differential-equation solvers. The research file notes that their integration cost is a bottleneck, and a closed-form survival model is enough for our purposes. MTM's coarse time bins also act as a constraint on our design. Diarization needs turn boundaries at the scale of tens of milliseconds, so we keep the frame-level acoustic encoder unchanged and apply irregular-time modeling only at the level of speaker memory, where gaps last seconds to minutes.
Research Program
The system keeps the streaming encoder of a Sortformer-style diarizer and replaces its speaker cache with a presence memory. Each memory slot $k$ stores three things. The first is an identity vector $e_k$, updated by running average only over frames confidently attributed to that speaker. The second is the time $s_k$, in seconds, when the speaker was last active. The third is a small history summary $c_k$ containing cumulative talk time, number of turns, and mean turn length. The probability that speaker $k$ is still present at time $t$ is modeled as a survival function:
$$p_k(t) = \exp\Big(-\int_{s_k}^{t} \lambda\big(e_k, c_k, u - s_k\big)\,du\Big),$$
where $\lambda \ge 0$ is a hazard rate, meaning the instantaneous rate of leaving the conversation. A small network produces $\lambda$ as a softplus over a few learned basis functions of elapsed time, so the integral has a closed form and no solver is needed. Frame-level activity for speaker $k$ is computed as the sigmoid of a similarity score between the frame embedding and $e_k$, shifted by a learned bias proportional to $\log p_k(t)$. In words, presence acts as a prior on activity and never changes the speaker's identity.
When $p_k(t)$ falls below a threshold, the slot moves to a dormant archive and its identity vector is kept intact. A newly detected voice is matched first against active slots and then against the archive, and only if neither matches is a new speaker created. The active set is bounded at size $A$, around four to six. The archive is unbounded, so the number of speakers in a session is unbounded too.
Presence is trained with a censored survival loss on re-entry times computed from ground-truth labels. For each silence gap, the target is the time until that speaker talks again, censored at the end of the session. This is combined with the usual frame-level diarization loss, and the arrival-order sort loss from Streaming Sortformer is kept for the active slots.
Training data comes from simulated long sessions built from LibriSpeech, with 2 to 16 speakers per session. Speakers enter and leave the session, and silence gaps follow a heavy-tailed distribution reaching several minutes. Real fine-tuning data comes from the DIHARD III and CALLHOME training portions.
Baselines
The naive reference point is Streaming Sortformer's own arrival-order cache, with eviction exactly as published. On DIHARD III, Streaming Sortformer reports 18.97% DER at 1.04 s latency, against 20.73% for EEND-GLA-Large. It is the strongest competitor because it is the only streaming system in the file that solves cross-chunk speaker tracking without matching, and our method changes only its memory. To keep the comparison fair, we train both systems from the same initialization, on the same simulated and real data, with the same number of steps, at the same 1.04 s and 0.32 s latencies. We run it twice: at its native four-speaker bound, and with the cache enlarged to 16 slots so that capacity is not the only difference. To count as beating it, our system must reach DER no worse than its result on DIHARD III and lower DER on sessions with more than four speakers.
The strongest offline references are AED-EEND, at 14.00% DER on full CALLHOME (against 15.43% for EEND-EDA) per AED-EEND, and TS-VAD at 7.6% DER on 8-speaker LibriCSS (against 14.9% for spectral clustering) per TS-VAD with improved i-vector estimation. Both see the whole recording, so we treat them as an upper bound that a streaming system aims to approach. We do not claim to beat them, and narrowing the gap to them on LibriCSS is the target. For CHiME-6 we report the numbers in the dinner-party TS-VAD paper for context only.
Four ablations, each trained under an identical budget, isolate the key ingredient. The first is a fixed-timeout eviction rule whose timeout is grid-searched on development data. The second feeds elapsed time to the cache as an ordinary input feature, with no survival structure and no identity/presence split. The third is TBM-style decay, which pulls the identity vector toward a population mean as silence grows. The fourth removes the hazard's dependence on $c_k$, so that leaving rates cannot differ between speakers.
Why not just…?
Why not just make the cache bigger? A 16-slot cache removes the capacity limit, but it keeps every speaker active forever. Each new voice then has more competitors to be confused with, and stale slots can produce false activations, the "ghost" behavior described for over-provisioned separators in the auxiliary-autoencoding paper. We run this enlarged cache directly. If it matches our method on sessions with many speakers and long gaps, the presence model is unnecessary, and we will report that result.
Why not just evict speakers after a tuned silence timeout? This is the most serious threat. A single timeout captures the main effect of elapsed time. The question is whether leaving rates actually differ across speakers and conversation types, for example between a chair who talks rarely but stays and a passer-by who says one thing and goes. We test this by breaking results down by speaker role and session type. If the tuned timeout comes within 0.3 DER points of our method on every split, we will drop the learned hazard in favor of the timeout and narrow our claim to the value of the identity/presence split with an archive.
Why not just re-identify returning speakers offline by clustering their embeddings? Re-clustering after the fact helps an offline system. A streaming system has to commit to labels at a latency of about one second, and clustering pipelines already trail neural diarizers on overlapped speech: 19.43% against 15.29% DER on CALLHOME in EEND-EDA. We include a streaming variant that adds a cosine-threshold archive match without any presence model. This isolates whether the gain comes from the archive alone.
Why not just treat dropped packets with zero-filling and dropout augmentation? For dropout robustness specifically, this may work as well as our approach, and we include it as the baseline for the dropout experiment. Our expectation is narrower. Augmentation alone does not tell the memory that physical time has passed, so after long dropouts it should handle re-entry worse. If it does not, we report that time-aware memory adds nothing for dropouts.
Proposed Experiments
The first decisive check plots DER and speaker-count error against the true number of speakers per session, from 2 to 16, on simulated long sessions and on the 8-speaker LibriCSS sessions. The prediction is that the Sortformer baselines degrade beyond four speakers while our curve stays flat. A flat curve for the 16-slot baseline as well would falsify the need for presence modeling.
The second check introduces a re-entry confusion rate: the share of returning speech, after silences longer than 30 s, 120 s, and 300 s, that receives the wrong speaker label or a new one. We measure it on DIHARD III and on the simulated sessions, comparing against the timeout and elapsed-time-feature ablations. The prediction is a gap that widens with silence length. We also expect the identity-decay ablation to perform worst at long gaps, which would test our reading of the Temporal Belief Memory tension.
The third check drops 10–40% of 100 ms audio packets in bursts and compares our method with zero-fill plus augmentation, focusing on DER around speaker returns that occur after the bursts.
Throughout, we report memory size and real-time factor next to the 0.093 that Streaming Sortformer reports, to show that the archive and hazard network add negligible cost. The research file reports no prior diarization results under packet loss and no re-entry-specific metrics. These two measurements would therefore be new benchmarks in their own right, whichever way the comparisons come out.
References
- Temporal Belief Memory: Imputing Missing Data during RNN Training
- End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractors
- Attention-based Encoder-Decoder Network for End-to-End Neural Speaker Diarization with Target Speaker Attractor
- Target-Speaker Voice Activity Detection with Improved i-Vector Estimation for Unknown Number of Speaker
- Separating Varying Numbers of Sources with Auxiliary Autoencoding Loss
- Streaming Sortformer: Speaker Cache-Based Online Speaker Diarization with Arrival-Time Ordering
- Serialized Output Training for End-to-End Overlapped Speech Recognition
- Continuous-time Autoencoders for Regular and Irregular Time Series Imputation
- Neural Continuous-Discrete State Space Models for Irregularly-Sampled Time Series
- MTM: A Multi-Scale Token Mixing Transformer for Irregular Multivariate Time Series Classification
- Target-Speaker Voice Activity Detection: A Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenario