Introduction
Standard Neural Machine Translation (NMT) typically operates in an "offline" mode, requiring the model to process an entire source sentence before generating a translation. This creates significant latency, making it impractical for real-time applications such as live subtitling or simultaneous interpretation. Simultaneous Machine Translation (SMT) addresses this by translating "on-the-fly," but it introduces a critical trade-off: translating too early risks inaccuracy due to lack of context, while waiting too long results in unacceptable delays.
Existing SMT strategies often rely on fixed rules, such as the "wait-k" method, or require extensive manual tuning. This paper introduces RLST (Reinforcement Learning for on-line Sequence Transformation), an agent that autonomously learns the optimal timing for reading input and writing output through trial and error.
The RLST Framework
The core of RLST is a Recurrent Neural Network (RNN) that functions as an interpreter agent. The translation process is modeled as a Partially Observable Markov Decision Process (POMDP), where the agent makes decisions based on its internal memory (the RNN state) without knowing future input tokens.
Action Space and Learning
At each step, the agent chooses between two primary actions:
- READ: Ingest the next token from the source sequence.
- WRITE: Generate a translated token based on the current context.
The model is trained using Deep Recurrent Q-Learning. It receives a negative reward for mistranslations and a specific penalty for attempting to READ after the input has concluded. A key innovation is the dynamic weighting of losses: early training focuses on translation accuracy, while later stages prioritize the RL-based timing strategy once basic language generation is mastered.
Experimental Setup
The authors evaluated RLST against standard Encoder-Decoder and Transformer models using two major benchmarks:
- Tatoeba: A diverse collection of sentence pairs. The researchers specifically isolated a "Long Test" split (sentences > 22 tokens) to test robustness.
- IWSLT2014: A standard dataset for German-English and English-German translation tasks.
Results and Discussion
RLST demonstrates emergent behavior similar to professional human interpreters, naturally learning to maintain a buffer (typically reading about five words ahead of its output).
Performance on Long Sequences
The most significant finding is RLST's superiority in handling long, complex sentences. While Transformers often struggle with long-range dependencies due to global attention complexity, RLST's RNN-based memory maintains context more effectively. As shown in the results table, RLST significantly outperforms Transformers on the Tatoeba "Long Test" sets across various language pairs.
For instance, in the English-Russian (Long) task, RLST achieved a BLEU score of 11.28, nearly doubling the Transformer's 5.66. On standard-length sentences, RLST remains competitive with state-of-the-art offline models despite the constraints of online processing. Furthermore, RLST maintains linear computational complexity relative to sequence length, ensuring efficiency.
Conclusion
RLST offers a robust solution for simultaneous sequence transformation by automating the delay strategy through reinforcement learning. While it shows slightly lower performance on specific tasks like IWSLT2014 English-German, its ability to handle long-form content makes it a powerful tool for real-time applications. Future work may extend this architecture to other sequential domains, such as real-time audio processing or bioinformatics.
Summary updated