Anytime Commitment for Streaming Translation with Offline Models

Live captioning and speech translation systems increasingly wrap a strong offline model, one trained on complete sentences, in a loop that re-translates the growing input every few hundred milliseconds. Output quality is high, but earlier words keep changing on screen, a problem called flicker. The usual fix is to freeze words after a fixed rule: hide the last $k$ words, or allow only the last three words to change. A fixed rule treats an easy word the same as a German verb that cannot be translated until the end of the clause. Coding theory has faced the same tension between waiting and being right. Block codes deliver nothing until a whole block arrives. Anytime codes instead emit an estimate of every past symbol immediately and let its error probability fall with elapsed delay. We propose to treat each displayed token the same way. The system estimates, for every emitted token, the probability that it will still be revised given how much new source has arrived since it appeared. A token is committed, meaning frozen for display and released to downstream consumers such as speech synthesis, once that probability falls below a target $\varepsilon$ chosen by the user.

In two sentences: flicker controls today are fixed windows, but how long a word stays unstable depends on the word and on its context. We learn a small delay-conditioned "revision hazard" model on top of an unchanged offline translator and commit each token when its predicted chance of future change drops below a budget. That turns an arbitrary window size into a calibrated error rate that should hold across language pairs.

Core Claim

Three claims are testable. First, how long a token stays revisable, measured in source units elapsed since it was generated, can be predicted from signals the wrapped model already produces. Second, a commitment rule based on that prediction beats every static window, mask, or hold rule on the joint quality–latency–flicker frontier, and the gain is largest on reordering-heavy directions. Third, calibrating the rule on held-out talks gives a controllable bound on the rate of committed-token errors, which is the quantity that matters when output cannot be retracted, for example when it is spoken aloud.

The anytime-reliability literature supplies the lens and a parametric hypothesis. Linear Error Correcting Codes with Anytime Reliability shows that simple causal tree codes achieve error decaying as $e^{-E d}$ in delay $d$. Anytime Reliable Codes for Stabilizing Plants over Erasure Channels shows why the exponent $E$, not the average error, determines whether a downstream process stays usable. We do not claim translation is a channel code. We borrow the idea that reliability should be a per-symbol function of delay, and we test empirically whether revision hazards actually decay exponentially.

Relation to Prior Work

The motivating evidence comes from the flicker literature. Re-Translation Strategies For Long Form, Simultaneous, Spoken Language Translation re-translates growing ASR transcripts with an offline system and curbs flicker with biased beam search plus a uniform Mask-$k$. The authors themselves call Mask-$k$ a blunt instrument. Improving Stability in Simultaneous Speech Translation: A Revision-Controllable Decoding Approach bounds revisions to the last $N$ tokens and finds $N=3$ a good compromise, but explicitly calls for adaptive revision windows because static ones fail under long-distance reordering. Both supply the baseline mechanisms. We replace their uniform window with a per-token, delay-dependent commit time.

The closest method to ours is PsFuture. It decides whether to read more input or write a token by measuring the cosine distance between the model's next-token distributions with and without a pseudo-future suffix. It is append-only and never measures flicker. We reuse its divergence signal as one feature, but in a different role. PsFuture asks "should I write now," once, at generation time. We ask "has this already-written token survived long enough," repeatedly, as source arrives, and we keep revision available until commit.

Incremental Blockwise Beam Search and its IWSLT system CUNI-KIT adapt wav2vec 2.0 and mBART without retraining and commit through Local Agreement or Hold-$n$. Once committed, tokens can never change. Local Agreement is effectively a one-bit, two-step survival test, and our hazard model generalizes it to a calibrated multi-step one.

On the systems side, LLM as Effective Streaming Processor shows that separating source and target position groups lets batch-trained LLMs append to the key-value cache without re-encoding. It was tested only under fixed wait-$k$. We use it to re-decode only the uncommitted tail, so revision cost scales with the tail rather than the whole history.

Simulstream supplies long-form evaluation with Normalized Erasure (NE, the number of displayed tokens deleted per final token) and real-time factor. It also constrains us: its voice-activity-detection segmentation cuts flicker about 8× but lowers quality. That is the block-code behaviour we aim to avoid.

Finally, the coding-systems work motivates the framing rather than supplying method. Tambur replaces block error correction with sliding-window streaming codes in videoconferencing and cuts freezes 78% against a within-frame block code. It is evidence that moving from block to sliding-window recovery pays off in practice. It also warns that dependencies downstream of recovery can still stall output, and the analogue here is a committed token whose meaning a later clause reverses.

Research Program

Measuring revision hazards. We run unconstrained re-translation with open offline models (SeamlessM4T and Canary through Simulstream for speech; an LLM with group position encoding for text) on development talks. We log each target token's full display history: when it first appeared, how many source frames or words arrived before it was last changed, and whether it survived to the final output. From these logs we estimate an empirical survival curve $S(d)$, the fraction of tokens still subject to future revision after $d$ source units. We break the curve down by language pair, part of speech, and alignment distance. This step answers a question nobody has measured: whether revision risk decays exponentially, as anytime reasoning suggests, or has a heavy tail for reordered heads. A heavy tail would imply that any fixed window is structurally wrong.

Hazard model. For a token $i$ at current delay $d$, a small classifier predicts

$$h_i(d) = \Pr(\text{token } i \text{ is revised after delay } d \mid \text{features}_i(d)).$$

The features are the token's current and initial model probability, the PsFuture divergence at the token's position, the number of consecutive re-decodes in which it has survived unchanged, the gap between its source-attention position and the latest source frame, and $d$ itself. We compare a free-form classifier with an exponential form $h_i(d)=\exp(-E_i\,(d-d_{0,i}))$, where the per-token exponent $E_i$ and onset $d_{0,i}$ are predicted from the features. The exponential form needs little calibration data and yields interpretable exponents. The translation model is never fine-tuned.

Commit rule and calibration. We commit token $i$, together with everything before it, at the first delay where $h_i(d)<\varepsilon$. The threshold is recalibrated on held-out talks with a split-conformal procedure: we choose the smallest threshold whose observed committed-token revision rate on calibration talks stays at or below $\varepsilon$. This gives a marginal guarantee that roughly an $\varepsilon$ fraction of committed tokens would have been revised had revision been allowed. Uncommitted tokens remain on screen and may change. Only committed tokens reach downstream speech synthesis.

Efficient revision. Committed prefix states stay cached. Using group position encoding for LLMs, and prefix-constrained decoding for encoder–decoder models, each update re-decodes only the uncommitted tail. We report real-time factor to show the approach avoids the quadratic cost of full re-translation.

Baselines

The naive reference is unconstrained re-translation with standard beam search. It gives the best final quality and the worst flicker, and it anchors both ends of the stability axis. Its opposite is revision-free decoding: fixed wait-$k$ for the LLM, and revision window zero for speech models. Improving Stability in Simultaneous Speech Translation reports that this setting reaches NE 0.00 and cuts German-to-English latency from 1057 ms to 689 ms, at a cost in BLEU.

The strongest static competitor is the revision-window method from the same paper. It reports BLEU 26.33 with NE reduced from 1.49 to 0.11 on CoVoST 2 German-to-English at $N=3$. That number comes from a model trained on proprietary data, so it cannot be compared with ours directly. We therefore re-implement revision windows on our open models and sweep $N$ from 0 to 8 per language pair, which gives the static method an oracle-tuned window per pair. That is more tuning than our method gets, since we use one $\varepsilon$ calibrated on generic held-out talks.

We also run biased beam search with Mask-$k$ from Re-Translation Strategies, which reports NE falling from 2.11 to 0.12 on English-to-German with BLEU within 0.3 of baseline. We sweep both the bias and $k$. The strongest onlinization competitor is Incremental Blockwise Beam Search with Local Agreement and Hold-$n$. It reports gains of 0.6 to 3.6 BLEU at fixed latency over standard blockwise search, and we run it with the same models and chunk sizes. For PsFuture, the file gives no numeric results: no reported result found on Lacuna for PsFuture BLEU/AL values. We re-run it append-only with its divergence threshold swept.

All methods share the same checkpoints, audio chunking, beam size, and GPU. Every method gets a hyperparameter sweep of equal size on the same development talks.

Three ablations isolate the key ingredient. The first is a single-feature commit rule that thresholds token probability at generation time, calibrated to the same $\varepsilon$. The second removes delay and survival-count features from the hazard model. The third replaces the exponential form with the free-form classifier.

We count the proposal as beating the baselines only under three conditions. At matched NE and matched LAAL, it must exceed the best per-pair-tuned revision window and Incremental Blockwise Beam Search on COMET. The gain must be clearly larger on German-to-English than on English-to-Romance directions. And the realized committed-error rate must track $\varepsilon$ on unseen language pairs.

Why not just…?

Why not just tune the static revision window or Mask-$k$ separately for each language pair? This is the most natural fix, and we give it the oracle-tuned sweep described above. It cannot adapt within a pair: a German sentence with a clause-final verb needs a long window at the verb and none elsewhere. The hazard measurements show directly whether survival varies more within a pair than between pairs. If the per-pair optimum matches our frontier, we will report that adaptivity within a pair is unnecessary.

Why not just commit tokens whose model probability exceeds a calibrated threshold? This is the real threat, because calibration alone already gives it an error-rate knob. The difference is that generation-time probability is a snapshot, whereas revision is caused by source that has not yet arrived. The survival count and the source-attention gap capture how the token has fared since. The single-feature ablation settles the question. If it matches the full model, the contribution reduces to calibrated confidence commitment, and we would say so.

Why not just segment with voice-activity detection and translate each segment offline? Simulstream shows it reduces flicker and compute but lowers quality. It is a block code: continuous speech without pauses yields no output at all. We include it as a baseline and report worst-case silence duration alongside the average-latency measures, where this failure appears.

Why not just use Local Agreement? Local Agreement commits after two consecutive agreeing re-decodes. That is one fixed survival test regardless of token, so it either waits too long on stable words or commits reordered heads too early. Running Incremental Blockwise Beam Search with Local Agreement on the same models, and plotting the committed-error rate of each method against its latency, shows which effect dominates.

Proposed Experiments

Speech experiments use MuST-C English-to-German and English-to-French long-form talks through Simulstream, and CoVoST 2 German-to-English as the reordering stress case. Text experiments use IWSLT-17 English-to-German with an LLM served using group position encoding. Metrics are BLEU and COMET, LAAL and AL via SimulEval conventions, NE, real-time factor, and a new committed-error rate. The committed-error rate is the fraction of committed tokens that are absent from or misplaced in the final unconstrained translation; it is the quantity that matters when output is fed to speech synthesis.

The first decisive check is descriptive: whether per-token survival curves fit an exponential with a varying exponent, or show a heavy tail concentrated on reordered verbs. The second is the frontier comparison at matched NE and latency. The third tests the calibration claim: a threshold $\varepsilon$ fit on English-to-French talks must hold within a factor of two on German-to-English without refitting. Failure there would mean hazards are pair-specific, and the method would need per-pair calibration data.