Fixed Latency Scheduling for Simultaneous Translation

60 concepts50 papers1 proposal

In traditional machine translation, a model waits to process an entire source sentence before generating its translation. While effective for offline text, this "full-sentence" paradigm fails in real-time scenarios like live subtitling, international conferences, or streaming speech-to-speech translation. To translate concurrently with an ongoing audio or text stream, a system must solve the Read/Write dilemma: it must decide exactly when it has enough context to translate accurately (Write) and when it must pause to consume more input (Read).

Early attempts to solve this relied on complex, dynamic policies driven by reinforcement learning or heuristics to predict the optimal moments to read or write. However, a surprisingly simple, deterministic alternative has emerged as a foundational strategy in simultaneous machine translation (SiMT): fixed-latency scheduling, most prominently realized as the wait-$k$ policy.

The Mechanics of the Wait-$k$ Schedule

The wait-$k$ policy solves the scheduling problem by replacing dynamic decision-making with a strict, predictable mathematical progression.

Instead of evaluating the information density of the incoming stream at each step, a wait-$k$ system establishes a fixed context buffer at the start of a sentence and maintains a constant lag thereafter. The policy operates via two simple rules:

  1. Initial Delay: The system reads exactly $k$ source tokens before generating the first target token.
  2. Interleaved Alternation: For every subsequent source token read from the stream, the system writes exactly one target token.
Source Stream:  [S1] -> [S2] -> [S3] -> [S4] -> [S5] ...
Wait-k (k=2):   (Read)  (Read)  (Write) (Read)  (Write)
Target Stream:                  [T1]            [T2]

As formalized in the seminal STACL framework by Mingbo Ma et al. in STACL: Simultaneous Translation with Implicit Anticipation and Controllable Latency using Prefix-to-Prefix Framework, this means that when generating the $t$-th target token, the model has access to exactly $t + k - 1$ source tokens. A closely related variant, the static read-write ($STATIC-RW$) policy proposed by Fahim Dalvi et al. in Incremental Decoding and Training Methods for Simultaneous Translation in Neural Machine Translation, similarly utilizes a predefined block of initial reads followed by a consistent, structured alternation pattern.

By operationalizing the schedule deterministically, fixed-latency policies eliminate the need to train a separate, complex controller to manage timing. The latency becomes a controllable hyperparameter ($k$): a lower $k$ minimizes delay but reduces context, while a higher $k$ improves translation quality at the cost of higher latency.

Why It Works: Implicit Anticipation

The core mystery of fixed-latency scheduling is how it handles structural divergence between languages. For example, when translating from a verb-final language (like German) to a verb-initial language (like English), a wait-$k$ model with a small $k$ is frequently forced to output an English verb before the German verb has even been spoken or typed.

How can a model translate what it has not yet seen? The answer lies in prefix-to-prefix training.

Rather than training on full sentence pairs and applying the wait-$k$ constraint only during inference, the model is trained from scratch to map source prefixes to target prefixes. During training, the attention mechanism is explicitly constrained: the target token at position $t$ is only allowed to attend to the first $t + k - 1$ source tokens.

This training constraint forces the model's internal representations to do something remarkable: implicitly anticipate the future. Because the network is penalized during training for incorrect early outputs, it learns to exploit statistical regularities in language to "hallucinate" or predict upcoming source concepts based on the partial context it has already received. For instance, if the partial German prefix describes a subject ordering food at a restaurant, the model learns to anticipate the final verb ("gegessen" / eaten) and generates the corresponding English verb early.

Vulnerabilities of Rigid Scheduling

While the simplicity of fixed-latency scheduling makes it highly robust, its rigidity introduces specific failure modes that highlight the trade-offs of the approach.

Target-Side Over-Reliance and Hallucination

Because a wait-$k$ model is frequently forced to generate tokens when source information is scarce, it faces a severe risk of compounding errors. As analyzed by Kehai Chen et al. in On the Hallucination in Simultaneous Machine Translation, when the prefix lacks sufficient semantic clues, the model shifts its attention away from the source stream and over-relies on its own previously generated target tokens. This self-reinforcing loop often causes the model to "hallucinate" fluent but entirely unsupported text or fall into repetitive loops.

The Cost of Monotonicity

Human interpreters do not use a fixed lag; they dynamically adjust their delay based on the grammatical complexity of the sentence. A fixed wait-$k$ policy treats a straightforward, word-for-word translation identical to a complex sentence requiring massive structural reordering.

To bridge this gap, researchers often contrast fixed schedules with adaptive ones. For example, Shaolei Zhang et al. introduced Gaussian Multi-head Attention for Simultaneous Machine Translation, which uses predicted word alignments as a soft prior. This allows the system to dynamically wait until the most informative source tokens are present, offering a more flexible trade-off than rigid token counting.

Scaling and Modern Adaptations

Despite its limitations, the simplicity of the wait-$k$ policy has allowed it to scale seamlessly from early recurrent neural networks to modern, large-scale architectures and continuous streaming applications.

From Sentences to Continuous Streams

Standard wait-$k$ policies assume text is already segmented into neat sentences, resetting the context buffer at every period. In real-world streaming scenarios, however, input arrives as an unbounded, unsegmented stream. Javier Iranzo-Sánchez et al. addressed this in From Simultaneous to Streaming Machine Translation by Leveraging Streaming History by adapting the wait-$k$ policy to track global token positions across an entire session, allowing the fixed-latency trade-off to remain stable over hours of continuous translation.

Word-Level Synchronization

Standard neural machine translation operates on subword units (like BPE tokens). Applying a wait-$k$ policy at the subword level means the model might count a single long word as 3 or 4 units of delay, artificially accelerating the schedule and forcing a translation before the full semantic unit is read. To stabilize this, Kang Kim et al. proposed Enhanced Simultaneous Machine Translation with Word-level Policies, which constrains the Read/Write transitions to occur strictly at word boundaries, ensuring complete semantic units are processed.

Integration with Large Language Models

The fixed-latency paradigm has also proven vital for adapting autoregressive Large Language Models (LLMs) to streaming audio tasks. In SimulS2S-LLM: Unlocking Simultaneous Inference of Speech LLMs for Speech-to-Speech Translation, Keqi Deng et al. demonstrated that applying a test-time wait-$k$ policy to boundary-aware speech prompts allows massive speech-to-speech LLMs to achieve state-of-the-art translation quality while maintaining a strict, predictable real-time response.

Key Foundational Work

STACL: Simultaneous Translation with Implicit Anticipation and Controllable Latency using Prefix-to-Prefix Framework (2019) — contributed 2 concept elements to this direction.
Incremental Decoding and Training Methods for Simultaneous Translation in Neural Machine Translation (2018) — contributed 1 concept elements to this direction.
Re-Translation Strategies For Long Form, Simultaneous, Spoken Language Translation (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_wait_k_policy_52b87401,
  title = {wait-k policy},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/wait-k-policy/rdr_52b87401d5af47da9e33ef6ded9af104},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_wait_k_policy_52b87401_v1,
  title = {wait-k policy},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/wait-k-policy/rdr_52b87401d5af47da9e33ef6ded9af104/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake