Self-Regulated Interactive Sequence-to-Sequence Learning

Julia Kreutzer, Stefan Riezler

Introduction

In the field of Neural Machine Translation (NMT), human feedback is a double-edged sword: it provides the highest quality training signal but comes at a significant financial and temporal cost. Traditional interactive machine learning systems often treat all feedback as equal, typically requesting full corrections for selected sentences. This paper proposes a more nuanced approach, framing the learning process as a "self-regulated" system where the model acts as an intelligent student, strategically deciding when to seek expensive expert help and when to rely on cheaper, noisier signals.

The Self-Regulated Framework

The authors introduce a meta-learning setup consisting of two primary components: a Student (a standard Seq2Seq NMT model) and a Regulator (a neural network that monitors the student's performance).

Overview of the self-regulated learning loop, illustrating the interaction between the Seq2Seq model and various feedback types of differing costs.

For every training example, the regulator chooses one of four actions based on the input sentence and the student's current hypothesis:

  1. FULL: Request a complete human re-translation (highest cost).
  2. WEAK: Request "error markings," where a human simply identifies correct segments of the output (lower cost).
  3. SELF: Utilize self-supervision via cross-view training on the model's own output (zero human cost).
  4. NONE: Skip the example entirely to save resources.

Training via Reinforcement Learning

The regulator is trained using Reinforcement Learning (RL), where the reward function is defined as the ratio of quality improvement (measured by BLEU score) to the feedback cost. This objective incentivizes the regulator to find the optimal "cost-quality trade-off."

Interestingly, the regulator naturally converges on a strategy reminiscent of an $\epsilon$-greedy algorithm. It learns to allocate roughly 25% of its budget to "exploration" via expensive feedback to maintain steady progress, while "exploiting" cheaper weak signals or self-supervision for the remainder of the training process.

Key Results and Efficiency

The researchers evaluated the framework on German-to-English translation tasks across multiple domains, including TED talks (IWSLT2017) and literary prose (Books).

BLEU score improvement relative to cumulative cost, demonstrating that mixing feedback types (full/weak/self) achieves higher performance at lower costs than full supervision alone.

Massive Cost Savings

The self-regulated models achieved performance gains comparable to full human supervision but at a fraction of the cost—reducing human labor requirements by up to a factor of 23. As shown in the cost-efficiency curves, strategies that mix feedback types (full/weak/self) consistently outperform traditional active learning that relies solely on full corrections.

Robust Domain Transfer

A significant finding was the regulator's generalizability. A regulator trained on the TED talks domain was able to effectively manage a student's learning on the Books dataset without any additional meta-training. In the Books domain, "Weak" feedback proved particularly effective, acting as a regularizer against the often non-literal and noisy nature of full literary translations.

Limitations and Future Directions

While the results are promising, the study currently relies on simulated human feedback and costs. Real-world human variability may introduce new challenges for the regulator. Additionally, the current RL setup assigns credit to entire mini-batches, making it difficult to pinpoint which specific sentence triggered a learning breakthrough. Future work aims to expand the action space to include more diverse human-centric feedback, such as pairwise rankings or time-based cost metrics.

Summary updated

Abstract

Not all types of supervision signals are created equal: Different types of feedback have different costs and effects on learning. We show how self-regulation strategies that decide when to ask for which kind of feedback from a teacher (or from oneself) can be cast as a learning-to-learn problem leading to improved cost-aware sequence-to-sequence learning. In experiments on interactive neural machine translation, we find that the self-regulator discovers an 𝜖-greedy strategy for the optimal cost-quality trade-off by mixing different feedback types including corrections, error markups, and self-supervision. Furthermore, we demonstrate its robustness under domain shift and identify it as a promising alternative to active learning.
BibTeX
@inproceedings{kreutzer2019self,
  author = {Julia Kreutzer and Stefan Riezler},
  title = {{Self-Regulated Interactive Sequence-to-Sequence Learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=ljLD3ZRpUYJ},
}
Figure