Keep Spliced Speech Out of the Streaming Path

Speech recognizers increasingly come as one model that runs two ways. A streaming mode transcribes live audio while seeing only a fraction of a second of the future. A full-context (offline) mode waits for the whole utterance and reads it in both directions. Training both modes in one network makes each better, and the full-context mode is usually the one that improves the streaming mode. All of this evidence comes from languages with hundreds to hundreds of thousands of hours of transcribed audio. For most of the world's languages there are perhaps ten hours. The most effective cheap fix at that scale is cross-lingual splicing: build fake training audio for the target language by stitching together fragments of English recordings that sound like the right phonetic units. Splicing has only ever been tested on offline models.

We argue that splicing and streaming fit together badly, and that the dual-mode model already contains the fix. A spliced utterance is full of artificial joints where natural transitions between sounds are missing. A full-context model can smooth over a bad joint using the words on either side. A streaming model looking through a 160–320 ms window often has the joint right at the edge of what it can see. Our proposal is asymmetric data routing. Spliced audio trains only the full-context path. The streaming path never hears synthetic speech. It learns from real target-language audio, with labels where we have them and with the full-context path's output probabilities as soft targets where we do not, including on untranscribed target audio. The full-context path then acts as a filter that turns noisy synthetic supervision into clean targets on real speech, inside a single deployable model.

Core Claim

In a dual-mode recognizer trained on about 10 hours of target-language data, we make two testable claims. First, naively adding spliced data helps the full-context mode much more than the streaming mode, and the shortfall grows as chunks shrink. Second, routing spliced data only to the full-context path, while the streaming path learns from real audio through in-model distillation, recovers most of the streaming benefit without hurting the offline mode. If the first claim fails, so that streaming gains as much from splicing as offline does, the routing idea has no reason to exist. We will report that as the main result.

Grounding in Prior Work

Two papers supply the parts we reuse. UniSplice maps target text to HuBERT discrete units, which are clusters of self-supervised speech features that work as rough language-independent phonetic units. It then concatenates matching fragments from 960 hours of English LibriSpeech. With 10 hours of simulated Common Voice data it reports 20–30% relative word error rate (WER) improvement on Indo-European targets and about 15% on Turkish. On Frisian it reports 12.3% WER against 16.7% for a VITS neural text-to-speech baseline. Its models are all offline, and that is the gap we exploit. XLSR-Transducer turns the multilingual self-supervised encoder XLSR-53 into a streaming transducer, meaning a decoder that emits tokens frame by frame as audio arrives. It does this with chunked attention masks and "attention sinks," which keep attention on the first frames. It reports streaming within up to 1.5% absolute WER of non-streaming on 100-hour Common Voice subsets. That paper provides our backbone, but it never goes below 100 hours.

Two papers supply the dual-mode training recipe. Dual-mode ASR shares weights across modes, distills the full-context path's token probabilities into the streaming path, and keeps separate normalization statistics per mode. That last detail matters for us, because routing different data to each mode will move their activation statistics further apart. Joint Optimization with Multi-Decoder and KD shows that separate decoders per mode reduce interference. On CSJ it reports 2.6–5.3% relative character error rate reduction for streaming over a standalone streaming model. We adopt its branched decoders. Both papers distill on the same real data that both modes see. We distill on data the streaming path is allowed to see, using a teacher that has seen data the student is not allowed to see.

One paper constrains the design. DuRep gets dual-mode parity through distillation at 8.3 million hours, a scale that is unavailable here. That is why we start from an existing pretrained encoder and do not pretrain. Two further papers frame the target regime. FLEURS shows that multilingual encoders still have high error rates on under-represented language groups, and it provides small per-language training sets. SC-MoE is the only prior work that combines dual-mode recognition with multiple languages. It needs frame-level language-identification labels and works on two high-resource languages, so it does not address scarcity.

Research Program

The model is an XLSR-53 encoder with dynamic chunk masking (chunks of 160, 320, and 640 ms plus full context), attention sinks, per-mode normalization statistics, a transducer decoder for streaming, and an attention decoder for the offline mode. Let $D_r$ be the real labeled target data, $D_u$ untranscribed real target audio, and $D_s$ spliced audio. The training objective is

$$ \mathcal{L} = \mathcal{L}_{\text{off}}(D_r \cup D_s) + \mathcal{L}_{\text{str}}(D_r) + \lambda \, \mathrm{KL}\big(\mathrm{sg}[p_{\text{off}}] \,\|\, p_{\text{str}}\big)\big|_{D_r \cup D_u}. $$

Here $\mathcal{L}_{\text{off}}$ and $\mathcal{L}_{\text{str}}$ are the supervised losses of the two modes. $p_{\text{off}}$ and $p_{\text{str}}$ are their token posteriors, aligned through the shared encoder frames. $\mathrm{sg}$ stops gradients into the teacher, and $\lambda$ weights the distillation term. The streaming path never receives gradients from $D_s$. Whether the shared encoder should receive streaming-path gradients from spliced data at all is exactly what our ablations vary.

The analysis has three parts. The first measures the splice penalty for each mode. We render held-out target sentences both as real recordings and as spliced audio, and report each mode's WER on both versions as a function of chunk size. The second locates errors relative to splice joints. In spliced evaluation audio we count errors whose aligned span crosses a joint, separating joints that fall at a chunk edge from joints inside a chunk. The third tracks how far apart the two modes' normalization statistics drift under each routing scheme.

Baselines

The do-nothing reference is the XLSR-Transducer dual-mode recipe trained on the 10 hours of real target data alone, with the same chunk schedule and the same decoders. This shows what splicing adds at all.

The strongest existing competitor is naive symmetric splicing: UniSplice data fed to both modes of the same dual-mode model, with in-place distillation on all data. This is exactly what a practitioner would do by combining the two published recipes. The closest published offline number is UniSplice's Frisian WER of 12.3%. That figure was obtained with a different, offline architecture, so we reproduce UniSplice's own offline result with our encoder as a calibration point. We found no reported result on Lacuna for streaming recognition at 10 hours with spliced data, so the streaming baseline numbers will be ours. Every system gets identical $D_r$, $D_s$, $D_u$, donor corpus, encoder initialization, number of update steps, and an equal hyperparameter search budget, with $\lambda$ and the mixing ratio tuned per system on the development set.

The ablations isolate the routing choice. In the first, spliced data reaches the streaming path but not the offline path. In the second, distillation runs on spliced audio and not on real audio. In the third, distillation is dropped and routing is kept. In the fourth, $D_u$ is removed, which separates the gain from routing from the gain from using untranscribed audio.

We claim success if routing lowers streaming WER below naive symmetric splicing at 160 and 320 ms chunks on at least four of five languages, keeps offline WER within run-to-run noise of the symmetric baseline, and holds or improves emission latency measured as median and 90th-percentile word emission delay. "Run-to-run noise" means the spread across three seeds.

Why not just…?

Why not just train on the spliced data in both modes, since UniSplice already works? This is the main threat. If chunked encoders are not actually more sensitive to splice joints, symmetric training is simpler and just as good. The splice-penalty measurement settles this directly. If streaming and offline gain equally from splicing at every chunk size, we report that negative result and drop the routing claim.

Why not just smooth the splices, for example with crossfades at the joints? Smoothing is an orthogonal fix applied to the data, and it may shrink the artifact. We include crossfaded splicing as a variant under both symmetric and routed training. If symmetric training on crossfaded audio closes the gap, the cheaper fix wins and we will say so.

Why not just train a separate offline teacher and pseudo-label untranscribed audio for a standalone streaming model? Classic self-training is a serious competitor, and it may match us on streaming accuracy. We run it with the same data, an offline teacher trained on $D_r \cup D_s$, and compute matched to the combined cost of training both its models. What our approach adds is that one model serves both modes. Prior evidence from Dual-mode ASR and Joint Optimization suggests joint training also improves each mode. If self-training ties on accuracy, the contribution shrinks to the routing finding and the halved deployment footprint.

Why not just run Whisper or a large multilingual model offline over buffered audio? Models trained only offline collapse when decoded in streaming fashion. XLSR-Transducer reports AMI WER of 35.3% before streaming training and 17.1% after, at 320 ms chunks. FLEURS also shows these models remain weak on under-represented languages. We include a buffered offline-decoding reference, with its measured latency, for completeness.

Proposed Experiments

The main testbed follows UniSplice: simulated 10-hour Common Voice training sets for Frisian, Dutch, German, French, and Turkish, with English LibriSpeech 960h as donor and held-out Common Voice audio as $D_u$. Turkish is the cross-family case, where UniSplice already gains least. We expect the splice penalty in streaming to be largest there, and routing to help most. A second testbed uses FLEURS languages with small native training sets, scored by character error rate, to check that the conclusions hold outside Indo-European languages and Latin script.

The first decisive check is the splice-penalty curve. Plot the difference between spliced-audio WER and real-audio WER for each mode against chunk size. The routing hypothesis predicts a streaming penalty that grows as chunks shrink and an offline penalty that stays flat. The second check compares the streaming WER of routing against symmetric splicing at matched latency. The third is the joint analysis: routed training should specifically reduce errors at joints that fall on chunk edges. A sweep of labeled data size across 2, 5, 10, and 20 hours shows whether the effect fades as real data grows, which it should if routing compensates for scarcity.