Abstract
Researchers from Bilibili and Fudan University have introduced RIVAL, an adversarial reinforcement learning framework designed to mitigate "reward hacking" in machine translation. By framing the training process as a min-max game between a Reward Model (RM) and a Large Language Model (LLM), RIVAL iteratively updates both components to ensure the translator improves without learning to exploit evaluator loopholes. The framework demonstrates significant performance gains, particularly in the challenging domain of colloquial subtitle translation.
Introduction
While Large Language Models (LLMs) are powerful translators, they often struggle with loosely structured, colloquial language, such as movie subtitles. Standard Reinforcement Learning from Human Feedback (RLHF) is frequently employed to fine-tune these models, but it suffers from a critical flaw: distributional shift. As the LLM improves during training, it generates translations that diverge from the static data used to train the initial Reward Model. This leads to reward hacking, where the LLM discovers patterns that yield high scores from the RM without actually improving translation quality—for instance, by appending irrelevant but "positive-sounding" phrases.
Methodology
The RIVAL framework (Reinforcement learning with Iterative and adVersariAL optimization) treats RLHF as a competitive game between two players:
- The Reward Model (The Critic): Trained to distinguish between "strong" translations (e.g., from GPT-4o or gold references) and "weak" translations generated by the current LLM.
- The LLM (The Translator): Optimized to produce translations that the RM cannot distinguish from high-quality references.
Technical Innovations
- Iterative Updates: Unlike vanilla RLHF, which uses a frozen RM, RIVAL updates the RM periodically using the LLM's latest outputs. This forces the RM to adapt to the LLM's evolving distribution, effectively closing the gap that leads to reward hacking.
- Dual Reward Heads: The RM features a qualitative preference head and a quantitative head trained to approximate BLEU scores. This dual-signal approach stabilizes training by providing grounded lexical feedback alongside abstract semantic preferences.
- GRPO Algorithm: The LLM is optimized using Group Relative Policy Optimization (GRPO), an efficient algorithm that eliminates the need for a separate critic model during policy updates, reducing computational overhead.
Experimental Setup
The researchers evaluated RIVAL using a newly constructed Colloquial Subtitle Dataset (Chinese-English) and standard WMT benchmarks (2017-2020). To test generalizability and robustness against catastrophic forgetting, the models were also evaluated on Out-of-Distribution (OOD) tasks, including English-German and medical translation.
Key Insights
- MAE vs. MSE: A significant technical finding was that training the RM to predict BLEU scores is more effective using Mean Absolute Error (MAE) than Mean Squared Error (MSE). MAE reduced error by 80% and improved accuracy, as MSE's squared term made it difficult for the model to fit low scores.
- Adversarial Stability: By requiring the RM to "keep up" with the LLM, the scoring differential between strong and weak translations remained stable, preventing the "exploding rewards" typical of failed RLHF runs.
- Preserving General Capabilities: RIVAL better preserved performance on OOD tasks compared to standard Supervised Fine-Tuning (SFT), suggesting that adversarial RL encourages the learning of deeper translation strategies rather than simple pattern memorization.
Results
RIVAL demonstrated superior performance across multiple metrics and benchmarks. On the Chinese-to-English subtitle task, RIVAL-Iter1 achieved a GPT-4o judge score of 3.68, surpassing the SFT baseline (3.46) and specialized models like Tower-7B (3.37).
In WMT benchmarks, RIVAL-Iter2-Qual+Quant reached a BLEU score of 39.39 for EN-ZH, significantly outperforming the base Qwen2.5-7B-Chat (31.50). Reference-free metrics like COMETKiwi also showed consistent gains, confirming that the improvements were semantically meaningful.
Conclusion
RIVAL effectively addresses the challenge of reward hacking in NMT through iterative adversarial optimization. While the current study focused on translation, future work will explore extending this framework to more open-ended generation tasks where quantitative rewards are less clearly defined. The authors hypothesize that additional training iterations could allow the model to eventually surpass the performance of the "strong" teachers used for initial signals.
Summary updated