Simultaneous Translation with Flexible Policy via Restricted Imitation Learning

Baigong Zheng, Renjie Zheng, Mingbo Ma, Liang Huang

Abstract

Simultaneous translation is a critical yet challenging NLP task that requires generating translations in real-time as source text arrives. Traditional approaches rely on either rigid fixed-latency policies or complex, unstable reinforcement learning (RL) frameworks. We propose a simplified single-model approach that incorporates a special “delay” token into the target vocabulary. By utilizing a restricted dynamic oracle to guide training through imitation learning, our model achieves flexible, adaptive policies. Experimental results on Chinese-English tasks demonstrate that this method yields superior translation quality (BLEU) and lower latency compared to both fixed and RL-based baselines.

Introduction

Simultaneous translation is often considered a "holy grail" of natural language processing, essential for real-time applications like international conferences. The primary hurdle is the structural divergence between languages, such as word order differences. For instance, translating from a verb-final language (like German) to a verb-initial one (like English) requires the system to decide whether to wait for the verb or attempt a guess.

Current methodologies generally fall into two categories:

  1. Fixed Policies: Systems like "wait-$k$" wait for a set number of source tokens before alternating between reading and writing. While simple, they lack the flexibility to adapt to varying sentence structures, often leading to premature guesses or unnecessary delays.
  2. Reinforcement Learning (RL): These models learn adaptive READ/WRITE policies. However, RL is notoriously difficult to train, typically requiring a multi-stage process where a translation model is pre-trained and then frozen while an RL agent is trained on top of it.

This research introduces a "middle way" that combines the simplicity of supervised learning with the flexibility of adaptive policies.

Methodology: Restricted Imitation Learning

The proposed approach transforms a standard Transformer Neural Machine Translation (NMT) model into a simultaneous translator through two key innovations.

The "Delay" Token

A special token, $\langle \epsilon \rangle$, is added to the target vocabulary. During decoding, if the model predicts a standard word, it performs a "WRITE" action. If it predicts $\langle \epsilon \rangle$, it performs a "READ" action, pausing to receive the next source word. This allows the model to manage its own latency dynamically.

Restricted Dynamic Oracle

To train the model, the authors developed a restricted dynamic oracle that provides an "expert" policy for the model to imitate. This oracle defines a valid "latency band" on a grid representing the alignment between source and target words.

The restricted dynamic oracle defines a latency band between an aggressive bound $\alpha$ (preventing reckless guessing) and a conservative bound $\beta$ (preventing excessive delay).

The band is constrained by two parameters:

  • $\alpha$ (Aggressive Bound): Ensures the model has seen enough source context before committing to a translation, preventing "hallucinations."
  • $\beta$ (Conservative Bound): Prevents the model from waiting too long, ensuring the translation remains truly simultaneous.

The training process uses Restricted Imitation Learning, where the model learns to stay within this optimal band. A Sigmoid loss is employed instead of Softmax to allow the model to treat READ and WRITE actions as non-competing labels when both are valid according to the oracle.

Key Insights and Implementation

A significant advantage of this method is its test-time controllability. By adjusting a "temperature" constant ($t$) for the delay token during inference, users can tune the model to be more "eager" (lower latency) or "patient" (higher quality) without retraining.

Additionally, the researchers addressed a noise issue in the Transformer's attention mechanism. They found that retaining all previous delay tokens in the history cluttered the attention layers. Their solution was to remove these tokens from the history but provide the model with an explicit count of previous delays via a separate embedding, maintaining latency awareness without the noise.

Results

The adaptive policy consistently outperformed both fixed wait-$k$ and RL-based models. On the NIST Chinese-English corpus, the model achieved higher BLEU scores at comparable latency levels, measured by Average Lagging (AL).

Qualitative analysis highlights the "smarter waiting" behavior of the adaptive policy. As shown in the comparison below, fixed policies like wait-1 often make incorrect lexical choices because they are forced to translate before the relevant source information is available.

Comparison of translation policies on a Chinese-to-English example. The wait-1 policy incorrectly guesses "thanks" before hearing the full source, while the adaptive policy correctly waits for the word "response" (回应).

Conclusion and Future Work

By treating the READ action as a vocabulary token and using a restricted oracle for imitation learning, this work simplifies the development of high-performance simultaneous translation systems. Future research directions include dynamically estimating the source-to-target length ratio during inference and integrating the model with speech-to-text systems to handle real-world acoustic features and disfluencies.

Summary updated

Abstract

Simultaneous translation is widely useful but remains one of the most difficult tasks in NLP. Previous work either uses fixed-latency policies, or train a complicated two-staged model using reinforcement learning. We propose a much simpler single model that adds a “delay” token to the target vocabulary, and design a restricted dynamic oracle to greatly simplify training. Experiments on Chinese <-> English simultaneous translation show that our work leads to flexible policies that achieve better BLEU scores and lower latencies compared to both fixed and RL-learned policies.
BibTeX
@inproceedings{zheng2019simultaneous,
  author = {Baigong Zheng and Renjie Zheng and Mingbo Ma and Liang Huang},
  title = {{Simultaneous Translation with Flexible Policy via Restricted Imitation Learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=JIzmHQJxBlK},
}
Figure