Reinforcement Learning for on-line Sequence Transformation

Grzegorz Rypesc, Lukasz Lepak, Pawel Wawrzynski

Introduction

Standard Neural Machine Translation (NMT) typically operates in an "offline" mode, requiring the model to process an entire source sentence before generating a translation. This creates significant latency, making it impractical for real-time applications such as live subtitling or simultaneous interpretation. Simultaneous Machine Translation (SMT) addresses this by translating "on-the-fly," but it introduces a critical trade-off: translating too early risks inaccuracy due to lack of context, while waiting too long results in unacceptable delays.

Existing SMT strategies often rely on fixed rules, such as the "wait-k" method, or require extensive manual tuning. This paper introduces RLST (Reinforcement Learning for on-line Sequence Transformation), an agent that autonomously learns the optimal timing for reading input and writing output through trial and error.

The RLST Framework

The core of RLST is a Recurrent Neural Network (RNN) that functions as an interpreter agent. The translation process is modeled as a Partially Observable Markov Decision Process (POMDP), where the agent makes decisions based on its internal memory (the RNN state) without knowing future input tokens.

Action Space and Learning

At each step, the agent chooses between two primary actions:

  1. READ: Ingest the next token from the source sequence.
  2. WRITE: Generate a translated token based on the current context.
The RLST architecture utilizes an RNN-based agent to decide between READ and WRITE actions, optimizing the delay for simultaneous translation.

The model is trained using Deep Recurrent Q-Learning. It receives a negative reward for mistranslations and a specific penalty for attempting to READ after the input has concluded. A key innovation is the dynamic weighting of losses: early training focuses on translation accuracy, while later stages prioritize the RL-based timing strategy once basic language generation is mastered.

Experimental Setup

The authors evaluated RLST against standard Encoder-Decoder and Transformer models using two major benchmarks:

  • Tatoeba: A diverse collection of sentence pairs. The researchers specifically isolated a "Long Test" split (sentences > 22 tokens) to test robustness.
  • IWSLT2014: A standard dataset for German-English and English-German translation tasks.

Results and Discussion

RLST demonstrates emergent behavior similar to professional human interpreters, naturally learning to maintain a buffer (typically reading about five words ahead of its output).

Performance on Long Sequences

The most significant finding is RLST's superiority in handling long, complex sentences. While Transformers often struggle with long-range dependencies due to global attention complexity, RLST's RNN-based memory maintains context more effectively. As shown in the results table, RLST significantly outperforms Transformers on the Tatoeba "Long Test" sets across various language pairs.

Comparison of BLEU scores across different architectures and datasets highlights RLST's significant advantage on long-form (L) test sets.

For instance, in the English-Russian (Long) task, RLST achieved a BLEU score of 11.28, nearly doubling the Transformer's 5.66. On standard-length sentences, RLST remains competitive with state-of-the-art offline models despite the constraints of online processing. Furthermore, RLST maintains linear computational complexity relative to sequence length, ensuring efficiency.

Conclusion

RLST offers a robust solution for simultaneous sequence transformation by automating the delay strategy through reinforcement learning. While it shows slightly lower performance on specific tasks like IWSLT2014 English-German, its ability to handle long-form content makes it a powerful tool for real-time applications. Future work may extend this architecture to other sequential domains, such as real-time audio processing or bioinformatics.

Summary updated

Abstract

A number of problems in the processing of sound and natural language, as well as in other areas, can be reduced to simultaneously reading an input sequence and writing an output sequence of generally different length. There are well developed methods that produce the output sequence based on the entirely known input. However, efficient methods that enable such transformations on-line do not exist. In this paper we introduce an architecture that learns with reinforcement to make decisions about whether to read a token or write another token. This architecture is able to transform potentially infinite sequences on-line. In an experimental study we compare it with state-of-the-art methods for neural machine translation. While it produces slightly worse translations than Transformer, it outperforms the autoencoder with attention, even though our architecture translates texts on-line thereby solving a more difficult problem than both reference methods.
BibTeX
@inproceedings{rypesc2022reinforcement,
  author = {Grzegorz Rypesc and Lukasz Lepak and Pawel Wawrzynski},
  title = {{Reinforcement Learning for on-line Sequence Transformation}},
  booktitle = {FedCSIS},
  year = {2022},
  url = {https://openreview.net/forum?id=RGMxlLwdXw},
}
Figure