Self-Regulated Interactive Sequence-to-Sequence Learning

Julia Kreutzer, Stefan Riezler

Abstract

Not all types of supervision signals are created equal: Different types of feedback have different costs and effects on learning. We show how self-regulation strategies that decide when to ask for which kind of feedback from a teacher (or from oneself) can be cast as a learning-to-learn problem leading to improved cost-aware sequence-to-sequence learning. In experiments on interactive neural machine translation, we find that the self-regulator discovers an $\epsilon$-greedy strategy for the optimal cost-quality trade-off by mixing different feedback types including corrections, error markups, and self-supervision. Furthermore, we demonstrate its robustness under domain shift and identify it as a promising alternative to active learning.

Introduction: The Cost of Human Feedback

In human-in-the-loop machine learning, human time is the most expensive resource. Traditional interactive systems often face a binary choice: request a high-cost, high-signal full correction or rely on zero-cost, low-signal self-learning. However, feedback exists on a spectrum. A teacher might provide a full correction, point out specific errors, or simply validate a correct output. This paper addresses the lack of "self-regulation" in models—their inability to determine when and what kind of feedback is most cost-effective for their current learning state.

Methodology: The Self-Regulated Learner

The proposed framework introduces a dual-component system consisting of a Student (Seq2Seq) model and a Regulator meta-learning module.

Conceptual diagram of the self-regulated interactive learning loop showing the interaction between the model and costly human feedback.

For each training instance, the Regulator evaluates the source sentence and the Student's hypothesis to select one of four feedback actions:

  • FULL: High-cost full human correction.
  • WEAK: Lower-cost binary error markings.
  • SELF: Zero-cost self-supervision (via cross-view training).
  • NONE: Zero-cost skip.

The Regulator is optimized using Reinforcement Learning (REINFORCE), with a reward function defined by the ratio of the Student's quality improvement to the feedback cost. This objective incentivizes the discovery of the most efficient learning signals.

Key Results and Efficiency

The self-regulated approach demonstrates significant advantages over traditional supervised and active learning methods. On the IWSLT (TED Talks) dataset, the system achieved translation quality comparable to full supervision while reducing human effort costs by a factor of 23.

BLEU score vs. Cumulative Cost on the IWSLT dataset, comparing various feedback mixing strategies against full supervision.

As shown in the results, strategies that mix feedback types (e.g., "full/weak/self") reach high BLEU scores much more efficiently than relying solely on full corrections. The regulator naturally discovers an optimal mixing ratio, balancing exploration of expensive signals early in training with exploitation of cheaper, effective signals as the model matures.

Insights and Domain Robustness

A notable finding is the system's performance in the "Books" domain. Here, the Regulator actually outperformed full human supervision. The authors attribute this to the nature of literary translation; human "corrections" in books are often non-literal and stylistic, which can introduce noise. In contrast, "weak" feedback—pointing out specific correct words—provides a more stable learning signal for the model.

Furthermore, the Regulator proved robust under domain shift. A Regulator trained on TED talks successfully managed a Student learning to translate literature, indicating that the meta-learned policy for feedback selection generalizes across linguistic styles.

Conclusion

Self-regulated learning offers a powerful alternative to standard active learning. By treating the choice of feedback as a learnable strategy, models can navigate the trade-off between supervision quality and human cost, achieving high performance with drastically reduced resources. Future work may explore more diverse feedback types and explicit credit assignment within training batches.

Summary updated

Abstract

Not all types of supervision signals are created equal: Different types of feedback have different costs and effects on learning. We show how self-regulation strategies that decide when to ask for which kind of feedback from a teacher (or from oneself) can be cast as a learning-to-learn problem leading to improved cost-aware sequence-to-sequence learning. In experiments on interactive neural machine translation, we find that the self-regulator discovers an $\epsilon$-greedy strategy for the optimal cost-quality trade-off by mixing different feedback types including corrections, error markups, and self-supervision. Furthermore, we demonstrate its robustness under domain shift and identify it as a promising alternative to active learning.
BibTeX
@inproceedings{kreutzer2019self,
  author = {Julia Kreutzer and Stefan Riezler},
  title = {{Self-Regulated Interactive Sequence-to-Sequence Learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=ljLD3ZRpUYJ},
}
Figure