Meeting transcription systems first pull overlapping voices apart with a neural network, a step called continuous speech separation. Because they listen in short overlapping windows of a couple of seconds, they must also decide, at every window boundary, which output channel continues which speaker's voice. Getting that "stitching" wrong swaps two people mid-sentence, even when each window on its own is separated well.
Early exit means stopping a deep network at an intermediate layer once some internal signal says the answer is good enough. It has already halved compute for separation on microphone arrays. Its confidence signals, however, look only inside the current window and ignore the stitching decision. They have also been reported to fail outright on single-microphone audio. This note proposes an exit rule that uses something overlapping-window systems compute anyway. The previous window has already committed an output for the stretch of audio the two windows share. That committed output is a free reference for checking the current window's intermediate estimate. The network stops only when its intermediate estimate agrees with that reference and the best speaker-to-channel match clearly beats the alternative.
Core Claim
The claim is that measured agreement on the shared audio with the previous window's committed output, together with how decisively one speaker-to-channel assignment wins, is a better stopping signal for streaming separation than predicted confidence or layer-to-layer stability. "Better" here means two things. At equal average compute it should give fewer speaker swaps and a lower speaker-attributed word error rate. It should also find exitable windows in single-microphone audio where the layer-stability criterion of Don't shoot butterfly with rifles finds almost none.
The signal is observed on the actual input rather than predicted by a head trained on clean data. The threshold brittleness under distribution shift noted in the early-exit literature should therefore be smaller. The Right Tool for the Job shows that intermediate confidences are miscalibrated without temperature correction.
Research Program
Two prior systems define the starting point and its limits. The early-exit work above runs a 16-layer Transformer on 7-channel LibriCSS. It attaches a small mask estimator to every layer and exits when the masks of adjacent layers differ by less than a threshold. It reports a 2x average speedup with matching or better word error rate, and a 6.59x speedup with the best word error rate when exiting at layer 2 on non-overlapped audio. It also reports that the scheme "did not translate well" to single-channel separation, and that its exit decision never consults stitching. Knowing When to Quit, called PRESS here, supplies the method we reuse for the new part of each window. Its exits output a Student-t distribution over the clean signal. It stops when the predicted signal-to-noise ratio clears a target with stated probability. It only exits when every speaker stream qualifies. It was evaluated only on pre-segmented clips and has no notion of streaming or stitching.
Our separator follows the sliding-window, double-buffered design of Low-Latency Speaker-Independent Continuous Speech Separation. It processes 2.4-second windows that overlap their predecessor. It has a 16-layer backbone with a lightweight per-layer head that outputs two speaker streams and a PRESS-style variance. Training uses depth-weighted per-layer losses with permutation-invariant training (Yu et al.).
At layer $l$ of window $t$, let $O$ be the audio shared with window $t-1$, and let $\bar s_{t-1,k}$ be the committed output of channel $k$ on $O$. For each assignment $\pi$ of current streams to previous channels, define the agreement
$$A_l(\pi)=\frac{1}{|K_a|}\sum_{k\in K_a}\mathrm{SISNR}\big(\hat s^{(l)}_{t,\pi(k)}|_O,\ \bar s_{t-1,k}\big).$$
Here $\hat s^{(l)}_{t,j}$ is stream $j$ estimated at layer $l$. $K_a$ is the set of channels whose committed signal on $O$ exceeds an energy floor. SISNR is scale-invariant signal-to-noise ratio. Let $\pi^\star$ be the best assignment and $\pi_2$ the runner-up, so the margin is $m_l=A_l(\pi^\star)-A_l(\pi_2)$.
The window exits at layer $l$ when three conditions hold. First, $A_l(\pi^\star)\ge a$ and $m_l\ge\delta$. Second, $\pi^\star$ has not changed since layer $l-1$. Third, the PRESS predicted signal-to-noise ratio on the non-shared part of the window clears its target. When $K_a$ is empty, the assignment does not matter and only the PRESS condition applies.
The agreement test has one known weakness: if the previous window was wrong, it confirms the error. We guard against this by forcing a full-depth pass whenever a run of consecutive early exits exceeds a budget. The forced pass re-anchors the reference. With two or three streams, scoring every assignment costs a handful of SISNR computations on about a second of audio, which is negligible next to a Transformer layer.
The analysis has two parts. First, we test whether exit-induced errors in single-channel separation are mainly within-window separation errors or mainly assignment errors at boundaries. This explains the reported failure rather than routing around it. Second, we test whether the margin $m_l$ predicts boundary swaps better than mask stability or predicted signal-to-noise ratio does.
Baselines
The naive reference is the full-depth model with standard boundary stitching. It is the identical network run with threshold zero. The early-exit work reports that multi-exit training slightly hurts the deepest layer, so we also train a vanilla single-exit 16-layer model to measure that cost. Static depth gets a second reference: uniformly shallow 4-, 6- and 8-layer models trained on the same data. The proposal must beat them on its Pareto curve at matched average multiply-accumulate operations.
The layer-stability criterion of Don't shoot butterfly with rifles runs on our backbone with its normalized mask distance and depth-weighted training. Its file-reported result is a 2x speedup at matching word error rate on 7-channel LibriCSS. No reported result was found on Lacuna for single-channel use, beyond the authors' statement that it failed.
The strongest competitor is PRESS applied per window with standard stitching. It uses the same variance heads, the same pessimistic all-speakers rule and the same stitching. The only difference from ours is the absence of the agreement and margin terms. It reports about 24.3 dB SI-SNRi on WSJ0-2mix, but only on clips. No reported result was found on Lacuna for PRESS on LibriCSS or any continuous-separation benchmark, so we establish that number ourselves.
The hard-bypass system of Overlap Aware Continuous Speech Separation without PIT skips separation when an overlap detector says one speaker is active. It reports cutting compute from 194.95G to 77.36G multiply-accumulate operations, with more than 1 dB better SNR than stitching-based separation, on simulated reverberant meetings. We reimplement its detector-gated bypass on our backbone. As a static, stitching-aware reference point we also report the file's LibriCSS figures for QPM: 14.1% word error rate at short overlap and 24.0% at long overlap.
Fairness rules apply to every system. All share the backbone, the training data, the same number of threshold-sweep points tuned on a common development split, and the same ASR back end.
Ablations isolate the key ingredient. Agreement only (no margin), margin only (no agreement), the rule without the PRESS term, and the rule without forced refresh separate the contribution of each piece.
The proposal beats the competition under two conditions, each at matched average compute. The first is a lower concatenated minimum-permutation word error rate and boundary swap rate than PRESS-per-window. The second, in the single-channel setting, is a mean exit depth well below 16 at word error rate within half a point of full depth, where layer stability stays near full depth.
Why not just…?
Why not just use PRESS per window and let ordinary stitching handle identity? This is the real threat. If its predicted signal-to-noise ratio already implies correct assignments, our extra terms add nothing. The ablation that drops agreement and margin settles this directly, and we will report a null result if they tie. Our reason to expect a gap is that PRESS's rule never sees the previous window. A window can be internally well separated while its two streams are nearly symmetric with respect to the shared audio, and that is precisely when a swap occurs.
Why not just gate with an overlap detector and skip separation on single-speaker audio? The bypass saves a lot of compute, but every detector miss on overlapped speech sends unseparated interference downstream with no fallback. A binary switch also cannot choose among the many intermediate depths that the 0%–40% overlap range of LibriCSS seems to call for. We compare the two systems stratified by overlap ratio and on detector-missed segments specifically.
Why not just train a smaller static model? PRESS reports that dynamic exits dominate static models of equal average compute on clips. Heavy reverberant overlap in LibriCSS remains hard: without separation the word error rate at 40% overlap is 43.3%, against 21.7% with full 7-channel processing (Continuous speech separation). The shallow models in our Pareto plot test whether this holds for streaming.
Why not just fix identity after separation with spatial or embedding clustering? Count And Separate shows that online direction-of-arrival tracking drifts over 10-minute sessions, with diarization error rising to 12.36% against 8.33% offline. Spatial tracking is also unavailable with a single microphone. Embedding tracking with long buffers, as in O-EENC-SD, adds a separate model and does not reduce separator compute. It is complementary rather than a substitute.
Proposed Experiments
Training uses simulated long-form meetings built as in the overlap-aware work: 3–5 speakers, high overlap, and reverberation times of 100–500 ms. The 7-channel WSJ1-derived mixtures of the early-exit work are added for the array condition. Single-channel versions take one microphone.
Testing uses LibriCSS in both single-channel and 7-channel form, stratified by overlap ratio. Metrics are word error rate and concatenated minimum-permutation word error rate from the Kaldi back end. We also measure boundary swap rate: the fraction of windows whose committed assignment disagrees with the oracle assignment computed from references. Compute is reported as mean exit layer and multiply-accumulate operations. As a streaming separation-quality check we evaluate 2-second-chunk Libri3Mix SI-SNRi against QPM's reported 18.2 dB.
The first decisive check replicates the single-channel failure of layer-stability exits. It then shows whether our rule recovers savings, and whether layer-stability's errors concentrate at boundaries.
The second check plots, per layer, how well three signals predict a boundary swap: margin $m_l$, mask distance, and PRESS predicted signal-to-noise ratio, scored by area under the ROC curve. The proposal is supported only if the margin is clearly the best predictor.
The third check tunes all thresholds on the low-reverberation development split and evaluates them on the high-reverberation split and on MUSAN-noised LibriCSS. This tests whether the observed-agreement signal holds its compute–accuracy operating point better than the predicted-confidence rules.
Finally, we vary the refresh budget to measure how often self-confirming errors accumulate, and what full-depth refresh costs to stop them.
References
- Don't Shoot Butterfly with Rifles: Multi-Channel Continuous Speech Separation with Early Exit Transformer
- The Right Tool for the Job: Matching Model and Instance Complexities
- Knowing When to Quit: Probabilistic Early Exits for Speech Separation Networks
- Low-latency Speaker-independent Continuous Speech Separation
- Permutation invariant training of deep models for speaker-independent multi-talker speech separation
- Overlap Aware Continuous Speech Separation without Permutation Invariant Training
- Quadruple Path Modeling with Latent Feature Transfer for Permutation-free Continuous Speech Separation
- Continuous Speech Separation: Dataset and Analysis
- Count And Separate: Incorporating Speaker Counting For Continuous Speaker Separation
- O-EENC-SD: Efficient Online End-to-End Neural Clustering for Speaker Diarization