Hearing aids, meeting transcribers and voice assistants all have to pull one known person's voice out of a long, messy recording as it arrives. The standard tool is target speech extraction: a network is given a short "enrollment" clip of the target speaker, turns it into a fixed voice fingerprint (a speaker embedding), and uses that fingerprint to filter the mixture. Two things break in long, live recordings. The fingerprint never changes even though the speaker's voice, distance and room keep changing. And when the target goes quiet, the extractor keeps producing something, usually leaked speech from someone else. The best published fix for the second problem runs a separate offline voice-activity detector before the extractor. Neither fix works in a causal, low-latency setting.
We propose to make the target's identity and presence part of the recurrent memory of a streaming separator. The backbone is SkiM, a low-latency LSTM separator that already passes a compact summary state between audio chunks. We add two things to that passed state: a target anchor, which is a running estimate of what the target sounds like right now, and a presence probability. The anchor is updated only when the model is confident the target is speaking alone. The presence probability both silences the output and freezes the anchor during target absence. The whole system trains end to end and runs causally. The claim we will test is narrow and falsifiable. On long sessions with enrollment mismatch and similar-sounding interferers, coupling anchor updates to learned presence should beat both a static fingerprint and a hand-built running average of re-extracted fingerprints, at SkiM-level compute.
Core Claim
Semi-blind separation is separation that uses partial prior knowledge about the sources. In this setting the prior is the enrollment clip, and the open question is how that prior should evolve over a stream. Existing extractors treat the prior as a constant input. We treat it as recurrent state with gated updates.
The specific mechanism is an anchor $a_k$ for chunk $k$, updated as
$$a_{k+1} = a_k + g_k \, p_k \, (u_k - a_k).$$
Here $u_k$ is a speaker embedding computed from the model's own extracted output for chunk $k$, $p_k \in [0,1]$ is the predicted probability that the target is active, and $g_k \in [0,1]$ is a learned update gate computed by the memory LSTM. The gate depends on how confident the model is, and on how far $u_k$ lies from the enrollment centroid and from embeddings of the other speakers in the stream.
The product $g_k p_k$ is the key ingredient. It stops the anchor from absorbing an interferer during target silence or confusion, which is the self-reinforcing failure that any naive re-enrollment scheme invites. The same $p_k$ scales the output mask, so absence handling and identity tracking share one causal state rather than living in two separately trained networks.
Relation to Prior Work
Three papers motivate the gap. Continuous Target Speech Extraction couples an attention-based target voice-activity detector with a personalized band-split RNN. It finds that filtering before extraction ("Cascade 1") is the only arrangement that prevents leakage, reaching 41.3 dB interference suppression, 26.5% diarization error and 10.4 dB SI-SNR. Its speaker embedding is static, the two modules are trained separately, and the authors leave joint training open. We do that joint training inside a causal model.
SC-TSE shows that pulling extracted audio toward an offline multi-utterance speaker centroid reduces speaker confusion: on Libri2Mix clean 100h, extraction accuracy rises from 91.08% to 95.15% and SI-SDR from 13.34 to 14.29 dB. It also shows that over-constraining identity hurts audio quality unless the loss is switched off past a threshold. We reuse its centroid-contrastive loss as a regularizer on the anchor, not on the output. This lets the anchor drift with the speaker without letting it cross into another speaker's region.
The Neural Target Speech Extraction overview names the silence problem and the cost of deep bidirectional LSTM extractors as unresolved obstacles to on-device use.
SkiM supplies the method. Its local LSTM processes short chunks, and a global memory LSTM passes only terminal hidden and cell states between chunks. This gives 17.1 dB SDRi in causal continuous separation at 2.0 G/s multiply-accumulates (versus 7.5 for dual-path RNN) and 0.92 ms CPU latency. SkiM is blind and relies on permutation-invariant training, so it does not know which output is the target across a long session. Its ablations also show that dropping either part of the passed state is catastrophic, which constrains our design: the anchor must be added to the passed state, not substituted for part of it.
Other informed-separation work fits the picture but differs in where the prior lives. MVAE uses a pretrained generative speech model as a prior but needs test-time optimization. PGSS tracks pitch recurrently and conditions a generator on it, but works offline on short utterances. CORSS updates an unmixing matrix online under a handcrafted prior, and its authors list adaptive priors as future work. All three keep the prior fixed or handcrafted. Ours learns when to change it.
Research Program
We will build the model on SkiM's causal configuration. The conditioning vector enters each chunk through feature-wise scaling and shifting (FiLM). In the static variant this vector is the enrollment embedding. In ours it is the current anchor $a_k$, which travels in the memory LSTM's state together with $h$ and $c$.
A lightweight frozen speaker encoder, the same one used for enrollment, embeds each chunk's extracted output to produce $u_k$. A small head on the memory state predicts $p_k$ and $g_k$.
Training combines four terms. The first is SI-SDR on target-active regions. The second is an energy penalty on output during target-absent regions, as in the interference-suppression metric. The third is binary cross-entropy on $p_k$ against ground-truth activity. The fourth is the SC-TSE centroid-contrastive term applied to $a_k$.
Training sessions are long simulated meetings built from LibriSpeech, following the SkiM meeting recipe and SparseLibriMix-style variable overlap. Enrollment is deliberately taken from a different LibriSpeech chapter than the session. Mid-session changes in room impulse response and gain create genuine drift.
Truncated backpropagation through time over 20–30 second windows makes end-to-end training tractable. Some memory states are carried across windows without gradient, so the model sees anchor states it did not initialize itself.
The analysis side asks three questions: when the gate opens, how far the anchor moves over a session, and whether anchor movement correlates with the injected acoustic changes rather than with interferer activity.
Baselines
The naive reference point is blind causal SkiM trained with permutation-invariant training, with no target information. Its output stream is chosen once at the start by embedding similarity to the enrollment. This measures how much any target information buys and exposes permutation swaps over long sessions. The reported number to beat on the meeting data is SkiM's causal 17.1 dB SDRi from SkiM. Because that figure is a blind, oracle-permutation number, our comparison rescores it under target-selection.
The key ablation is static-anchor informed SkiM: identical architecture, data and training budget, but $a_k$ fixed to the enrollment embedding and $p_k$ still used as an output gate. This isolates the anchor update. Our claim fails if the adaptive version does not beat it by a clear margin on drift and similar-speaker subsets, which we set before running as at least 1 dB SI-SDR on target-active regions and a reduction in speaker-confusion rate.
A second ablation removes the coupling by setting $p_k = 1$ in the anchor update while still gating the output. This tests whether absence-aware updating, not adaptation alone, is what matters.
The strongest hand-built alternative is static-anchor SkiM plus exponential-moving-average re-enrollment from the model's own output. We tune its rate and an optional similarity threshold on the same validation set with the same search budget as our gate hyperparameters.
The strongest existing system for long recordings is Cascade 1 from Continuous Target Speech Extraction, with reported 41.3 dB interference suppression, 26.5% diarization error and 10.4 dB SI-SNR on its continuous benchmark. It is non-causal and much heavier. We will evaluate on that benchmark's protocol and report the numbers alongside MACs and latency. "Beating" it means matching its interference suppression within a few dB and its SI-SNR or better while remaining causal at roughly SkiM-level compute. Outright superiority is not required for the claim, but we report it honestly either way.
On short utterances, the reference is SC-TSE's 14.29 dB SI-SDR and 95.15% extraction accuracy on Libri2Mix clean 100h (SC-TSE). Here we only expect parity, since short clips leave little room for drift.
All learned systems share training data, enrollment protocol, speaker encoder and GPU-hour budget.
Why not just…?
Why not just keep a static embedding and rely on the network's own recurrence to adapt? The LSTM state can in principle absorb drift implicitly, and with enough training this may suffice. The static-anchor ablation tests exactly this. If its gap to our model is under the preset margin on drift subsets, we conclude that explicit anchor state is unnecessary and report that.
Why not just re-enroll on the fly with a moving average of extracted embeddings? This is the most serious threat, because it is cheap and needs no new training. Its weakness is predictable. During target silence or confusion it averages in the interferer and then extracts the interferer more confidently, a feedback loop the static fingerprint never has. The tuned moving-average baseline and the uncoupled ablation, evaluated on the subset with high-similarity interferers and long target absences, settle this directly. If a well-tuned threshold version matches us, the honest conclusion is that the learned gate adds little, and the contribution reduces to joint presence gating.
Why not just run an external voice-activity detector before the extractor, as Cascade 1 does? Continuous Target Speech Extraction shows this cascade suppresses leakage well. However, its detector is non-causal, adds a second network, and fails in turn when its own static embedding confuses similar speakers, as the authors note. We compare against it directly and also build a causal cascade (a SkiM-sized presence detector feeding static-anchor SkiM) at matched compute. That separates the benefit of integration from the benefit of causality.
Why not just separate blindly and pick the target stream per chunk? The naive baseline, rerun with per-chunk reselection, covers this. Its expected failure is stream swaps and output during target silence, which the interference-suppression metric captures.
Decisive Experiments
The main test set is long simulated LibriSpeech meetings (90-second sessions, 50–80% overlap, noise and reverberation as in SkiM) with three stress factors crossed. The first is enrollment from a different chapter and room. The second is a mid-session room or gain change. The third is interferer similarity, binned by speaker-encoder cosine similarity to the target. We add the continuous target-extraction benchmark of Zhao et al. for comparison with Cascade 1, SparseLibriMix for the low-overlap regime where LibriMix reports that strong separators lose to simple masks, and Aishell2Mix for cross-lingual transfer.
We will report six metrics: SI-SDR on target-active regions, interference suppression on target-absent regions, speaker-confusion rate (fraction of chunks whose output embedding is closer to an interferer), diarization error from $p_k$, MACs, and CPU latency.
The decisive result is the interaction. The adaptive, presence-coupled anchor should widen its lead over both the static anchor and the moving-average baseline as session length, drift and interferer similarity increase, while staying level on short clean clips. We will also check that anchor displacement tracks injected acoustic changes and stays flat during target absence.
If the lead does not grow with these factors, the recurrent anchor is not doing what we claim, whatever the average scores say.
References
- Continuous Target Speech Extraction: Enhancing Personalized Diarization and Extraction on Complex Recordings
- SC-TSE: Speaker Consistency-Aware Target Speaker Extraction
- Neural Target Speech Extraction: An overview
- Skim: Skipping Memory Lstm for Low-Latency Real-Time Continuous Speech Separation
- Semi-blind source separation with multichannel variational autoencoder
- PGSS: Pitch-Guided Speech Separation
- Constrained online recursive source separation framework for Real-Time electrophysiological signal processing
- LibriMix: An Open-Source Dataset for Generalizable Speech Separation