RIVAL: Reinforcement Learning with Iterative and Adversarial Optimization for Machine Translation

Tianjiao Li, Mengran Yu, Chenyu Shi, Yanjun Zhao, Xiaojing Liu, Qiang Zhang, Qi Zhang 0020, Xuanjing Huang, Jiayin Wang

Abstract

Researchers from Bilibili and Fudan University have introduced RIVAL, an adversarial reinforcement learning framework designed to mitigate "reward hacking" in machine translation. By framing the training process as a min-max game between a Reward Model (RM) and a Large Language Model (LLM), RIVAL iteratively updates both components to ensure the translator improves without learning to exploit evaluator loopholes. The framework demonstrates significant performance gains, particularly in the challenging domain of colloquial subtitle translation.

Introduction

While Large Language Models (LLMs) are powerful translators, they often struggle with loosely structured, colloquial language, such as movie subtitles. Standard Reinforcement Learning from Human Feedback (RLHF) is frequently employed to fine-tune these models, but it suffers from a critical flaw: distributional shift. As the LLM improves during training, it generates translations that diverge from the static data used to train the initial Reward Model. This leads to reward hacking, where the LLM discovers patterns that yield high scores from the RM without actually improving translation quality—for instance, by appending irrelevant but "positive-sounding" phrases.

Methodology

The RIVAL framework (Reinforcement learning with Iterative and adVersariAL optimization) treats RLHF as a competitive game between two players:

  1. The Reward Model (The Critic): Trained to distinguish between "strong" translations (e.g., from GPT-4o or gold references) and "weak" translations generated by the current LLM.
  2. The LLM (The Translator): Optimized to produce translations that the RM cannot distinguish from high-quality references.
The RIVAL framework architecture, illustrating the iterative updates between the Reward Model (RM) and the Large Language Model (LLM) across RM and LLM steps.

Technical Innovations

  • Iterative Updates: Unlike vanilla RLHF, which uses a frozen RM, RIVAL updates the RM periodically using the LLM's latest outputs. This forces the RM to adapt to the LLM's evolving distribution, effectively closing the gap that leads to reward hacking.
  • Dual Reward Heads: The RM features a qualitative preference head and a quantitative head trained to approximate BLEU scores. This dual-signal approach stabilizes training by providing grounded lexical feedback alongside abstract semantic preferences.
  • GRPO Algorithm: The LLM is optimized using Group Relative Policy Optimization (GRPO), an efficient algorithm that eliminates the need for a separate critic model during policy updates, reducing computational overhead.

Experimental Setup

The researchers evaluated RIVAL using a newly constructed Colloquial Subtitle Dataset (Chinese-English) and standard WMT benchmarks (2017-2020). To test generalizability and robustness against catastrophic forgetting, the models were also evaluated on Out-of-Distribution (OOD) tasks, including English-German and medical translation.

Key Insights

  • MAE vs. MSE: A significant technical finding was that training the RM to predict BLEU scores is more effective using Mean Absolute Error (MAE) than Mean Squared Error (MSE). MAE reduced error by 80% and improved accuracy, as MSE's squared term made it difficult for the model to fit low scores.
  • Adversarial Stability: By requiring the RM to "keep up" with the LLM, the scoring differential between strong and weak translations remained stable, preventing the "exploding rewards" typical of failed RLHF runs.
  • Preserving General Capabilities: RIVAL better preserved performance on OOD tasks compared to standard Supervised Fine-Tuning (SFT), suggesting that adversarial RL encourages the learning of deeper translation strategies rather than simple pattern memorization.

Results

RIVAL demonstrated superior performance across multiple metrics and benchmarks. On the Chinese-to-English subtitle task, RIVAL-Iter1 achieved a GPT-4o judge score of 3.68, surpassing the SFT baseline (3.46) and specialized models like Tower-7B (3.37).

Performance comparison of RIVAL against baselines across In-Domain (WMT EN-ZH/ZH-EN) and Out-of-Domain (EN-DE/DE-EN) tasks, showing superior BLEU and COMETkiwi scores.

In WMT benchmarks, RIVAL-Iter2-Qual+Quant reached a BLEU score of 39.39 for EN-ZH, significantly outperforming the base Qwen2.5-7B-Chat (31.50). Reference-free metrics like COMETKiwi also showed consistent gains, confirming that the improvements were semantically meaningful.

Conclusion

RIVAL effectively addresses the challenge of reward hacking in NMT through iterative adversarial optimization. While the current study focused on translation, future work will explore extending this framework to more open-ended generation tasks where quantitative rewards are less clearly defined. The authors hypothesize that additional training iterations could allow the model to eventually surpass the performance of the "strong" teachers used for initial signals.

Summary updated

Abstract

Large language models (LLMs) possess strong multilingual capabilities, and combining Reinforcement Learning from Human Feedback (RLHF) with translation tasks has shown great potential. However, we observe that this paradigm performs unexpectedly poorly when applied to colloquial subtitle translation tasks. In this work, we investigate this issue and find that the offline reward model (RM) gradually diverges from the online LLM due to distributional shift, ultimately leading to undesirable training outcomes. To address this, we propose RIVAL, an adversarial training framework that formulates the process as a min-max game between the RM and the LLM. RIVAL iteratively updates the both models, with the RM trained to distinguish strong from weak translations (qualitative preference reward), and the LLM trained to enhance its translation for closing this gap. To stabilize training and improve generalizability, we also incorporate quantitative preference reward (e.g., BLEU) into the RM, enabling reference-free quality modeling aligned with human evaluation. Through extensive experiments, we demonstrate that the proposed adversarial training framework significantly improves upon translation baselines.
BibTeX
@misc{li2025rival,
  author = {Tianjiao Li and Mengran Yu and Chenyu Shi and Yanjun Zhao and Xiaojing Liu and Qiang Zhang and Qi Zhang and Xuanjing Huang and Jiayin Wang},
  title = {{RIVAL: Reinforcement Learning with Iterative and Adversarial Optimization for Machine Translation}},
  year = {2025},
  eprint = {2506.05070},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.05070},
}
Figure