Self-Enrolling Speaker Slots for Streaming Separation of an Unknown Number of Talkers

Live captioning, meeting transcription, and hearing aids all have to pull individual voices out of overlapping speech in real time. The people talking are strangers to the model, their number is unknown, and it changes as people join and leave. Current systems handle at most two of these conditions at once. Causal (streaming) separators assume a fixed number of speakers. Separators that count speakers need the whole recording in advance. Target-speaker extractors, which pull out one known voice, need that voice recorded ahead of time.

We propose a streaming separator that enrolls speakers itself, from the stream. When a new voice appears, the model opens a "slot", a small memory vector describing that voice. It fills the slot from the speaker's own clean stretches of speech as the conversation goes on. On every audio chunk it extracts one waveform per active slot. Speaker features come from a self-supervised speech model, meaning one pretrained on large amounts of unlabeled audio, converted to run causally. Because slots persist across chunks, output channels do not swap identities at chunk boundaries. Because slots are created on demand, there is no hard cap on how many speakers the system can handle.

Core Claim

Consider two sub-problems in streaming multi-talker separation: keeping each output channel tied to the same person across chunks, and handling an unknown and changing number of people. Both can be solved by one mechanism, a growing set of per-speaker memories that the model fills with its own separated output. This works better than the two usual approaches. One gives a fixed-output separator more outputs than it should ever need and stitches its windows together afterward. The other runs diarization (deciding who spoke when) first and then extracts each speaker in a separate step.

We make two testable predictions. First, separation quality for a new speaker should rise within the first few seconds after they begin talking, much as enrollment-based extraction improves with longer reference audio. Second, identity swaps over long sessions should be far rarer than with window-and-stitch systems. The self-supervised frontend matters because lower layers of such models carry speaker identity well, as Exploring wav2vec 2.0 on Speaker Verification and Language Identification shows. That paper reports 3.61% equal error rate on VoxCeleb1 after fine-tuning, against 24.28% when trained from scratch. Useful slots have to be built from only a second or two of possibly noisy speech, and features of this kind should make that possible.

Where This Sits

Three lines of work motivate the gap.

The closest system in spirit, Online Speaker Diarization of Meetings Guided by Speech Separation, does separate online. It fixes each 5-second window to 2 or 3 outputs and leaves speaker identity to later x-vector clustering. That is exactly the stitching design we argue against.

Two papers supply parts of the method.

One paper constrains the design. Listen to Extract: Onset-Prompted Target Speaker Extraction finds that prepending the enrollment waveform beats embedding-based conditioning offline, at 24.1 dB on WSJ0-2mix and 22.0 dB with 0.5 s of enrollment. Its authors also note that this prompting is hard to run online because attention cost grows quadratically with sequence length. We take the finding seriously and test a cached-prompt variant of their idea against compact slots, rather than assuming slots win.

Research Program

The system has four parts, each run chunk by chunk with a configurable lookahead of 0, 40, or 160 ms.

Causal frontend. We follow the distillation approach of the causal pretrained frontend paper, with a non-causal pretrained model such as WavLM or HuBERT as teacher. A causal student learns to reproduce the teacher's lower and middle layers. Lower-layer outputs feed the speaker memories. Middle-layer outputs feed the extractor.

Slot memory. Each slot $s$ holds a vector $m_s$. After each chunk, $m_s$ is updated as a running average of lower-layer features from frames that the model assigns to $s$ with high confidence and little overlap. The weight on a new frame is that frame's assignment confidence, so uncertain frames change the memory little.

Slot birth and retirement. An attractor head in the style of EEND-EDA looks at whatever part of the chunk the existing slots do not explain. It proposes candidate speakers, each with an existence probability. A candidate becomes a new slot only if its probability exceeds a threshold and it is sufficiently dissimilar from every existing slot, including dormant ones. A slot goes dormant after a stretch of silence. If a later candidate closely matches a dormant slot, that slot reactivates, which lets returning speakers keep their identity.

Extraction. A causal separator, conditioned on each active slot by feature modulation, produces one waveform per active slot. An activity head then gates each output, as EEND-SS does, so that silent slots do not emit leakage. Compute grows linearly with the number of concurrently active speakers. Dormant slots cost only a similarity check.

Training. We simulate sessions of 30 to 120 s with 1 to 6 speakers drawn from LibriSpeech. Speakers enter and leave at random, and overlap ranges from 0 to 100%. At training time we grow speaker density to the regime where EEND-EDA is reported to degrade, beyond 4 to 5 speakers.

The loss matches slots to reference speakers once per session, not once per chunk. Any identity swap therefore costs the model directly:

$$\mathcal{L} = -\max_{\pi} \sum_{j=1}^{J} \text{SI-SDR}\big(\hat{x}_{\pi(j)}, x_j\big) + \lambda \mathcal{L}_{\text{act}} + \mu \mathcal{L}_{\text{birth}}$$

The symbols are defined as follows:

  • $x_j$ is reference speaker $j$'s full-session waveform, and $J$ is the number of speakers in the session.
  • $\hat{x}_{\pi(j)}$ is the concatenated output of the slot assigned to speaker $j$ under the assignment $\pi$.
  • $\mathcal{L}_{\text{act}}$ is the activity-gating loss.
  • $\mathcal{L}_{\text{birth}}$ is the loss on slot existence probabilities.
  • $\lambda$ and $\mu$ are loss weights.

Slot memories begin training filled from oracle clean enrollment. On a schedule, they switch to being filled from the model's own estimates, so the model learns to recover from contaminated memories. This switch is the main risk to the design, and we analyze its failure modes explicitly.

Baselines

The naive reference is the unprocessed mixture, which gives 0 dB SI-SDRi by definition. A second naive point is a causal two-output separator with no pretrained frontend, run on every chunk. It measures how badly fixed capacity fails once more than two people talk.

The fixed-count streaming baseline is the causal ConvTasNet with a causal pretrained frontend from Causal Self-Supervised Pretrained Frontend With Predictive Patterns for Speech Separation. We extend it to $K_{\max}=6$ outputs. Unused outputs are trained against dummy silent targets and their loss is masked, following Target-Speaker Voice Activity Detection with Improved i-Vector Estimation for Unknown Number of Speaker. We add EEND-SS-style activity gating. Across chunks, outputs are linked by matching each output to the previous chunk's outputs through the frames the two chunks share. This is the most direct test of whether slots are needed at all. It shares our frontend, training data, lookahead, and tuning budget. We match parameter count and give both systems the same number of hyperparameter trials.

The strongest existing online competitor is the separation-guided online diarization system of Online Speaker Diarization of Meetings Guided by Speech Separation. It uses 5-second windows with 3 outputs and incremental x-vector clustering, and reports 27.2% DER on AMI against 28.5% for the prior online baseline. We retrain it on our simulated sessions and swap in our frontend, so the comparison isolates the stitching strategy rather than the features.

The strongest cascade runs EEND-GLA online diarization and then feeds each speaker's non-overlapped diarized segments into a streaming target-speaker extractor. That extractor uses compact embedding conditioning in the manner of Streaming Target-Speaker ASR with Neural Transducer. EEND-GLA reports 14.29% DER on CALLHOME and 20.42% on six-speaker simulated mixtures. This baseline differs from our system only in whether enrollment is learned jointly and refined continuously.

For offline upper bounds, the file gives these reference points:

System Setting Metric Reported value
EEND-SS Libri2Mix SI-SDRi 11.20 dB
EEND-SS 2/3-speaker mixtures Speaker counting accuracy 98.2%
A-DCSS 2 speakers, anechoic ASI-SDR 9.7 dB
A-DCSS Unknown count, noisy and reverberant ASI-SDR 7.7 dB

For streaming separation of a variable, unknown number of speakers: no reported result found on Lacuna.

Two ablations isolate the key ingredients. The first replaces the slot memories with fixed x-vectors computed from the first 2 s of each speaker, which removes continuous self-enrollment. The second replaces the causal pretrained frontend with learned convolutional filters trained from scratch, which removes the pretrained speaker features.

We count the proposal as successful only if it clearly beats the over-provisioned baseline and the cascade on session-level SI-SDRi, without exceeding their latency or real-time factor. The advantage must be largest in sessions with three or more speakers or with speakers who re-enter.

Why not just…?

Why not just give a causal separator six outputs and stitch the chunks? This is the real threat. When at most two or three people ever overlap, it may do as well as slots. Its known weakness is stitching. The meeting system above still needs x-vector clustering downstream to fix identities, and O-EENC-SD lists chunk-boundary permutation as the central difficulty. Session-level SI-SDRi and the identity-swap rate, broken down by session length and re-entry, will settle the question. If the baseline ties us on those, the slot mechanism is not worth its complexity, and we will report that.

Why not just run online diarization, then extract each speaker from its diarized clean segments? This cascade is our strongest baseline, and it is close to our design. The difference is that its enrollment is frozen once captured and never trained against the extractor. Integrated streaming systems have beaten cascades before: Streaming Target-Speaker ASR with Neural Transducer reports large gains over a streaming extraction-then-recognition cascade. That result comes from a different task, though, so we test it directly. The most telling measurement is quality in the first 1 to 3 s after a speaker starts. Short references are where global embeddings are known to fail, as reported by Fitting New Speakers Based on a Short Untranscribed Sample and by the content-dependent speaker embedding work of CDFSE, which measured about 3.10 similarity score for global embeddings against 3.51 for fine-grained ones.

Why not just prepend each speaker's harvested audio as a waveform prompt, as Listen to Extract does? In causal attention, a prompt can be cached, so its cost is not strictly quadratic per chunk. It still grows with prompt length and number of speakers, and the Listen to Extract authors themselves flag online use as impractical. We implement this variant with 2 s cached prompts per speaker and compare it at matched latency. If it wins on quality within the real-time budget, we adopt it as the slot representation.

Why not just use a non-causal pretrained model with a small lookahead? We sweep lookahead from 0 to 160 ms for all systems. The frontend choice is ablated separately from the slot mechanism, so the two effects are not confounded.

Proposed Experiments

Main evaluation. The main testbed is simulated variable-count sessions built from LibriSpeech, with entries, exits, re-entries, and 1 to 6 speakers. For each system we report:

  • session-level SI-SDRi;
  • identity-swap rate;
  • speaker counting accuracy;
  • delay from a speaker's first word to the birth of their slot;
  • real-time factor and algorithmic latency.

Standard benchmarks connect the results to prior numbers. On SparseLibriMix and Libri2Mix/Libri3Mix we compare with EEND-SS's offline figures. On the A-DCSS multi-utterance LibriSpeech noisy and reverberant condition, we compare with its 7.7 dB unknown-count result. The expected outcome is a streaming penalty. Streaming target-speaker ASR shows roughly 20% relative degradation versus offline, and our aim is a small, quantified gap rather than parity.

Zero-shot generalization. We train only on LibriSpeech and test on Vox2Mix and LRS2Mix-style sessions of unseen speakers in new acoustic conditions. The causal pretrained frontend paper showed its largest gains under exactly this kind of domain shift.

Real recordings. On REAL-M we measure word error rate after separation. On AMI we derive DER from slot activity and compare with the 27.2% online figure.

Decisive checks. Three curves decide whether the core claim holds.

  • Self-enrollment curve. Plot per-speaker SI-SDRi against time since that speaker's onset. It must rise and then level off, while the frozen x-vector ablation stays flat.
  • Density curve. Plot results as the number of concurrent speakers goes from 2 to 6. The advantage over the six-output baseline must grow as density increases.
  • Contamination analysis. Deliberately corrupt slot memories with another speaker's frames and measure recovery time. This shows whether the self-feedback loop corrects errors or amplifies them.

If the first curve is flat, or the third shows runaway contamination, the self-enrollment claim fails regardless of aggregate scores.