In simultaneous speech translation and streaming speech recognition, a model must output text while the speaker is still talking. The naive approach to this problem is re-translation: every time a new chunk of audio arrives, the model re-evaluates the entire audio history and generates a fresh hypothesis from scratch.
While re-translation allows the model to correct its past mistakes as more context becomes available, it introduces a severe user experience flaw: visual flickering or jitter (Papi et al.). Words that appeared on the screen suddenly vanish or change, causing cognitive fatigue for human readers and breaking downstream streaming modules like incremental text-to-speech (Fukuda et al.).
To prevent this "word salad" effect, streaming systems must decide exactly when a generated token is stable enough to be permanently committed (Cho et al.). This is the core purpose of Local Agreement decoding—a class of policies designed to verify hypothesis stability on the fly.
The Intuition of the Immortal Prefix
Why can we trust any part of a partial translation before a sentence is finished? The key insight lies in how autoregressive sequence-to-sequence models update their internal predictions over time.
When a model decodes a sequence based on an incomplete audio stream, the trailing edge of its prediction is highly volatile. Lacking future context, the model "fantasizes" endings prematurely (Niehues et al.). However, the beginning of the sequence quickly converges. As more audio chunks arrive, the top-ranked hypotheses generated by beam search begin to share an identical starting sequence.
The authors of Nguyen et al. conceptualize this as an immortal prefix. If a sequence of tokens remains unchanged despite the arrival of new acoustic data, it is highly unlikely to change in the future. If we can dynamically detect this stable prefix, we can safely display it to the user, knowing we will never have to retract it.
Audio Chunk t: [The cat sat on...] -> Hypothesis t: "The cat sat on the"
Audio Chunk t+1: [The cat sat on a...] -> Hypothesis t+1: "The cat sat on a mat"
^^^^^^^^^^^
Longest Common Prefix
(Safely Committed)
The Local Agreement Mechanism
How do we algorithmically isolate this immortal prefix? The most prominent framework is the Local Agreement (LA) policy (Ko et al.).
A major advantage of Local Agreement is that it operates entirely on the text outputs of consecutive decoding steps. This makes it highly versatile, allowing it to function even when internal model states—like cross-attention weights—are inaccessible, such as when prompting black-box Large Language Models (Macháček et al.).
The basic algorithm works as follows:
- Let $H_t$ be the full translation hypothesis generated at audio chunk $t$.
- Let $H_{t+1}$ be the hypothesis generated at the next chunk $t+1$.
- Compute the longest common prefix (LCP) between $H_t$ and $H_{t+1}$ (Tan et al.).
- Commit and display the tokens within this LCP, freezing them so they cannot be altered by future evaluations.
To make this strategy more robust against noise, the commitment criterion can be extended across a wider window. For instance, a token might only be finalized if it consistently appears in the top hypotheses across $k$ consecutive input increments (Polák et al.; Macháček et al.). By adjusting $k$, system designers can directly tune the balance between translation latency and output stability.
Balancing Quality and Latency
By freezing the committed prefix, the search space for future tokens becomes constrained. The authors of Polák et al. note that this incremental pruning prevents the model from retroactively fixing early translation choices, which can cause a minor drop in raw offline metrics like BLEU scores.
However, the practical benefits outweigh this trade-off. Local agreement completely eliminates visual jitter, drastically improving human comprehension in real-time applications. Furthermore, because this strategy can be applied as an "onlinization" wrapper during beam search, it allows researchers to convert highly accurate, pre-trained offline models into low-latency streaming systems without retraining them from scratch (Polák et al.).
Key Foundational Work
Punctuation insertion for real-time spoken language translation (2015) — contributed 1 concept elements to this direction.
Low-Latency Neural Speech Translation (2018) — contributed 1 concept elements to this direction.