Separating a recording of several overlapping talkers is often done by "peeling": a network pulls out the loudest voice, subtracts it, and repeats on what remains until nothing is left. Peeling handles an unknown number of speakers gracefully. Its cost is that the whole network runs once per speaker, one pass after another, and mistakes in early passes corrupt later ones. In a long meeting this cost is paid again for every short window of audio, even though the same people are usually talking in window $t$ as in window $t-1$. Video compression faced the same kind of waste decades ago and solved it with keyframes. A keyframe is a fully encoded picture. The frames between keyframes are coded only as changes from what came before.
This note proposes applying that idea to iterative separation. A full peel runs only at "keyframe" windows. In between, each already-known speaker is carried forward by a parallel, conditioned extraction step. Sequential peeling is then run only on whatever energy remains unexplained after the carried-forward speakers are subtracted. The bet is that per-window compute and latency then scale with the number of changes in the scene (people entering, leaving, or returning) rather than the number of active speakers. The second bet is that anchoring each speaker to its previous estimate reduces the deletion errors that audio-only stopping rules suffer in long windows.
Core Claim
Current iterative separators already exploit some continuity across time, but none exploit the redundancy of the extraction itself. The recurrent selective attention network RSAN (Continuous Speech Separation with Recurrent Selective Attention Network) peels speakers from each audio chunk of a meeting. It passes the leftover mask from one chunk to the next as a starting bias, and this lowered word error rate (WER) at 30% overlap on LibriCSS from 14.8% to 14.1%. Yet it still runs the full sequence of peeling passes in every chunk. It also reports that its stop decision becomes unreliable in 4.8 s windows, where quiet speakers are cut off early. We claim that treating the previous chunk's separated speakers as a prediction removes most sequential passes, and that reserving discovery for the leftover signal does three things. It turns the $N$-pass sequential loop into one batched pass plus a short loop over new arrivals. It makes the stop decision easier, because the question becomes "is anyone new here?" rather than "is anyone left?". It also keeps each speaker in a consistent output slot across the meeting.
Prior work plays three roles here. RSAN, the One-and-Rest recursive training of Recursive Speech Separation for Unknown Number of Speakers, and the counting-by-residual design of ReSepNet supply the peeling machinery, which we reuse unchanged as the "keyframe" step. Rethinking the Separation Layers in Speech Separation Networks motivates keeping single-output peeling rather than abandoning it, since it shows pure single-output iterative networks match multi-output ones (9.3 vs 9.0 dB SI-SDR). The video-efficiency literature supplies the design pattern. Compressed Video Action Recognition shows that keyframes plus coded changes carry the needed information at a fraction of the cost. KeyWorld shows that spending heavy computation only on transition-critical frames gives over 5× speedup. DyCoke shows that a "recycle bin" which lets dropped items be restored later avoids the irreversibility of one-shot pruning. None of these video works touches source separation. None of the separation works skips or amortizes extraction passes across time.
Research Program
The system keeps a set of tracked speakers. Each one stores a speaker embedding and the tail of its last estimate, taken from the overlap between consecutive windows.
When a new window arrives, the propagation step runs the same single-output extractor once per tracked speaker. Each run is conditioned on that speaker's embedding and overlap tail, and all runs are batched in parallel. The propagated estimates are subtracted from the mixture to form a residual.
The discovery step then applies the One-and-Rest presence test to this residual. Only if it fires does the system peel new speakers sequentially, exactly as RSAN would.
Tracked speakers whose propagated energy falls below a floor are moved to a dormant bin rather than deleted, in the spirit of DyCoke's recycle bin. When the discovery step finds a new source, it is first matched against dormant embeddings, so a returning speaker regains its old slot.
A refresh policy decides when to discard all tracks and run a full peel from scratch, bounding drift in the way periodic keyframes bound error in a codec. We compare three refresh rules: a fixed period, refresh triggered when residual energy after propagation exceeds a threshold, and a small learned trigger.
Per window, sequential network depth drops from roughly $N_t+1$ passes, for $N_t$ active speakers plus a final stop check, to
$$ D_t = 1 + A_t + 1, $$
where $A_t$ is the number of newly arriving speakers in window $t$. The first term is the single batched propagation pass and the last is the stop check. In conversational speech $A_t$ is zero in most windows, which is where the savings should come from.
Training uses simulated long-form streams in which speakers enter, pause, return, and leave. We use scheduled sampling: the propagation step is gradually fed its own previous estimates instead of oracle ones, so it learns to correct rather than amplify drift. The loss combines scale-invariant signal-to-distortion ratio (SI-SDR, a waveform fidelity measure insensitive to loudness) on each output with a binary cross-entropy for the presence test.
Baselines
The naive reference point is RSAN without block dependency, which re-peels each window independently. Its LibriCSS WER at 30% overlap is 14.8% (RSAN), and it fixes the compute ceiling: one sequential pass per speaker per window plus a stop pass.
The strongest existing competitor is RSAN with block-wise dependency. It is the only published method that already uses cross-window continuity inside a peeling loop. It reports 14.1% WER at 30% overlap and 14.5% WER on 3-speaker hot spots, against 17.6% for utterance-level permutation-invariant training (uPIT) (RSAN). We will reimplement it on the same extractor backbone, window lengths (2.4 s and 4.8 s), training streams, and tuning budget as our model, and use the same ASR back end.
The uPIT multi-output separator is a second reference for the parallel, fixed-channel approach, since it runs in one pass but leaks speech into empty channels.
Beating RSAN-with-dependency means two things together. First, WER no worse on every LibriCSS overlap condition and lower on hot spots and in the 4.8 s window, where RSAN's early stopping hurts. Second, at least a 2× reduction in sequential passes per second of audio, measured on the same hardware.
Two ablations isolate the key ingredient. The first removes discovery-on-residual: it propagates tracks but always re-peels fully. This tests whether savings come from skipping passes and not just from warm starting. The second removes propagation: it uses discovery only, with embeddings used for slot assignment after the fact. This tests whether conditioning on the previous estimate, rather than mere identity tracking, is what lowers deletions. A third ablation removes the dormant bin to measure its effect on speaker-consistent slots.
On ReSepNet's setting, the file gives 96.6% speaker-count accuracy (ReSepNet), but no reported SI-SDR was found on Lacuna for ReSepNet on WSJ0-kmix. We will therefore report its count accuracy as the counting reference and rerun it ourselves for SI-SDR.
Why not just…?
Why not just keep RSAN's block-wise dependency, which already carries state across windows? RSAN's carried state only biases the first pass. Every speaker is still re-extracted sequentially, so latency still grows with the number of speakers. RSAN's own long-window deletion problem shows that its stop decision is not stabilized by the carried mask. The direct comparison and the "always re-peel" ablation settle whether propagation adds anything beyond RSAN's bias.
Why not just use a multi-output separator with a generous channel budget, which runs in one pass? It cannot exceed its channel count. On LibriCSS it leaks speech into silent channels and degrades sharply in hot spots (17.6% vs 14.5% WER against RSAN). Rethinking the Separation Layers shows it is weakest at low overlap, which is the common case in meetings. Our propagation step recovers its main advantage, parallelism, without the fixed channel count.
Why not just diarize the stream and run target-speaker extraction for each known speaker? This is the real threat, because our propagation step is effectively conditioned target-speaker extraction. What diarize-then-extract lacks is the residual. A speaker the diarizer misses is never extracted, and there is no subtraction to reveal them. We found no reported result on Lacuna for diarization-plus-target-extraction on LibriCSS. We will therefore build this pipeline from the same extractor with an oracle-free embedding clusterer and compare deletion errors on streams where speakers enter mid-utterance. If it matches us, the contribution reduces to the residual discovery step, and the discovery ablation will tell us exactly how much that step is worth.
Why not just run a full peel every $k$-th window and copy assignments in between? This is the separation analogue of uniform frame subsampling. It saves compute but produces no fresh audio for the skipped windows, unless the speaker set is frozen and extracted anyway, which is our method with a fixed refresh period. The fixed-period refresh rule is therefore included as a baseline against the residual-triggered and learned refresh rules.
Proposed Experiments
The primary benchmark is LibriCSS, the real-meeting continuous separation set of RSAN. We report WER per overlap ratio, WER on 3-speaker hot spots, insertion and deletion breakdowns, sequential passes per second, and real-time factor at 2.4 s and 4.8 s windows.
The second benchmark is a long-form stream built by concatenating and overlapping WSJ0-2mix and WSJ0-3mix utterances with scripted entries, pauses, and returns. We train only on streams with at most three simultaneous speakers and test on streams built from the WSJ0-4mix and WSJ0-5mix speaker counts, following the generalization protocol of ReSepNet. Metrics are SI-SDR improvement, per-window count accuracy, slot-consistency (the fraction of a speaker's speech assigned to a single output), and $A_t$-normalized compute.
A reverberant two-speaker check on the LibriSpeech simulation of Rethinking the Separation Layers verifies that the propagation step does not cost fidelity in the easy case. That paper's 9.3 dB SI-SDR pure-SISO figure serves as the sanity floor.
Three checks are decisive. The first is the drift curve: SI-SDR as a function of windows since the last refresh. Under scheduled sampling this curve should be flat or slowly declining. A steep decline would mean propagation compounds error over time, as early peeling errors do within a window. The second plots compute against the number of speaker changes per minute. Our method should scale with changes and RSAN with speakers, and if the two curves coincide the core claim fails. The third is deletion error in the 4.8 s window. This is where an anchored "is anyone new?" test should clearly outperform RSAN's "is anyone left?" test. If it does not, the stopping benefit we claim is not real.
As a constrained extension, the refresh trigger can take a visual input on LRS3-based multi-talker streams. We would use frame-to-frame token similarity of face crops, as in DyCoke's temporal merging, to signal that a new face has begun moving its lips. This is reported only if the audio-only trigger leaves a measurable deletion gap.
References
- Continuous Speech Separation with Recurrent Selective Attention Network
- Recursive Speech Separation for Unknown Number of Speakers
- ReSepNet: A Unified-Light Model for Recursive Speech Separation with Unknown Speaker Count
- Rethinking The Separation Layers In Speech Separation Networks
- Compressed Video Action Recognition
- KeyWorld: Key Frame Reasoning Enables Effective and Efficient World Models
- DyCoke: Dynamic Compression of Tokens for Fast Video Large Language Models