Introduction
In the field of Neural Machine Translation (NMT), human feedback is a double-edged sword: it provides the highest quality training signal but comes at a significant financial and temporal cost. Traditional interactive machine learning systems often treat all feedback as equal, typically requesting full corrections for selected sentences. This paper proposes a more nuanced approach, framing the learning process as a "self-regulated" system where the model acts as an intelligent student, strategically deciding when to seek expensive expert help and when to rely on cheaper, noisier signals.
The Self-Regulated Framework
The authors introduce a meta-learning setup consisting of two primary components: a Student (a standard Seq2Seq NMT model) and a Regulator (a neural network that monitors the student's performance).
For every training example, the regulator chooses one of four actions based on the input sentence and the student's current hypothesis:
- FULL: Request a complete human re-translation (highest cost).
- WEAK: Request "error markings," where a human simply identifies correct segments of the output (lower cost).
- SELF: Utilize self-supervision via cross-view training on the model's own output (zero human cost).
- NONE: Skip the example entirely to save resources.
Training via Reinforcement Learning
The regulator is trained using Reinforcement Learning (RL), where the reward function is defined as the ratio of quality improvement (measured by BLEU score) to the feedback cost. This objective incentivizes the regulator to find the optimal "cost-quality trade-off."
Interestingly, the regulator naturally converges on a strategy reminiscent of an $\epsilon$-greedy algorithm. It learns to allocate roughly 25% of its budget to "exploration" via expensive feedback to maintain steady progress, while "exploiting" cheaper weak signals or self-supervision for the remainder of the training process.
Key Results and Efficiency
The researchers evaluated the framework on German-to-English translation tasks across multiple domains, including TED talks (IWSLT2017) and literary prose (Books).
Massive Cost Savings
The self-regulated models achieved performance gains comparable to full human supervision but at a fraction of the cost—reducing human labor requirements by up to a factor of 23. As shown in the cost-efficiency curves, strategies that mix feedback types (full/weak/self) consistently outperform traditional active learning that relies solely on full corrections.
Robust Domain Transfer
A significant finding was the regulator's generalizability. A regulator trained on the TED talks domain was able to effectively manage a student's learning on the Books dataset without any additional meta-training. In the Books domain, "Weak" feedback proved particularly effective, acting as a regularizer against the often non-literal and noisy nature of full literary translations.
Limitations and Future Directions
While the results are promising, the study currently relies on simulated human feedback and costs. Real-world human variability may introduce new challenges for the regulator. Additionally, the current RL setup assigns credit to entire mini-batches, making it difficult to pinpoint which specific sentence triggered a learning breakthrough. Future work aims to expand the action space to include more diverse human-centric feedback, such as pairwise rankings or time-based cost metrics.
Summary updated