Introduction
Machine Translation (MT) has seen significant advancements with the rise of Large Language Models (LLMs). However, standard training paradigms often struggle with the nuances of colloquial speech and the phenomenon of "reward hacking." RIVAL (Reinforcement learning with Iterative and adVersariAL optimization) is a novel framework designed to address these challenges by treating the Reward Model (RM) and the LLM as opponents in an iterative, adversarial min-max game. By combining qualitative preference ranking with quantitative metric approximation, RIVAL significantly outperforms standard fine-tuning and vanilla RLHF, particularly on challenging, loosely structured colloquial subtitles.
The Challenge: Exposure Bias and Reward Hacking
While LLMs are powerful translators, standard Supervised Fine-Tuning (SFT) often suffers from exposure bias. In this scenario, errors accumulate because the model only sees "perfect" training data during training but must navigate its own imperfect outputs during inference. Reinforcement Learning from Human Feedback (RLHF) is the typical remedy, but it often fails on colloquial subtitles due to distributional shift.
In vanilla RLHF, the Reward Model is typically trained once (offline). As the LLM improves, it enters a data distribution the RM has not encountered. Eventually, the LLM learns to "hack" the reward—generating text that satisfies the static RM's criteria (e.g., sounding polite or using specific keywords) while ignoring the source text's actual meaning. This results in high reward scores that do not correlate with actual translation quality.
The RIVAL Framework
RIVAL introduces an adversarial approach inspired by Generative Adversarial Networks (GANs) to ensure the Reward Model remains a robust teacher. The framework is built on three pillars:
- Adversarial Min-Max Game: Training is formulated as a competition. The RM (the discriminator) is trained to distinguish between "strong" translations (sourced from gold references or high-tier models like GPT-4o) and "weak" ones generated by the current LLM. Simultaneously, the LLM (the generator) is trained to close this quality gap.
- Iterative Updates: Rather than using a static RM, RIVAL updates both the LLM and the RM in successive rounds. In each iteration, the LLM generates new translations, which are then used to re-train the RM. This ensures the "teacher" evolves alongside the "student."
- Multi-Head Reward Model: To stabilize training, the RM utilizes two heads. One head ranks translations qualitatively (preference), while the second head is trained to approximate quantitative metrics like BLEU. This acts as a safety rail, ensuring lexical fidelity while the qualitative head focuses on semantic nuances.
For the RL optimization step, RIVAL utilizes Group Relative Policy Optimization (GRPO). This efficient algorithm computes advantages within a group of samples, removing the need for a separate, computationally expensive value model.
Experimental Setup and Datasets
The researchers evaluated RIVAL across several demanding benchmarks:
- Colloquial Subtitle Dataset: A new benchmark created from real-world Bilibili video subtitles using Automatic Speech Recognition (ASR). This dataset is characterized by informal, diverse, and loosely structured language.
- WMT (World Machine Translation): Standard benchmarks for ZH-EN and EN-ZH were used to test performance on formal language.
- Out-of-Distribution (OOD) Sets: To test generalization, the models were evaluated on medical translations (ZH-DE) and other language pairs (EN-DE, DE-EN).
Key Insights and Results
The "Online" Teacher
The most critical finding is that the Reward Model must be a "moving target." An offline RM eventually becomes a "sitting duck" for reward hacking. By updating the RM iteratively, RIVAL maintains a consistent pressure on the LLM to improve actual translation quality rather than just optimizing for a static metric.
Optimization Nuances
The authors discovered that when training the RM to approximate BLEU scores, Mean Absolute Error (MAE) is significantly more effective than Mean Squared Error (MSE). Because reward values are often small, MSE’s squared term can cause gradients to vanish. MAE provided a consistent signal that reduced error by 80% compared to MSE.
Performance Gains
- Subtitle Task: RIVAL-Iter1 achieved a GPT-4o evaluation score of 3.68, significantly higher than the SFT baseline of 3.46.
- WMT Benchmarks: On the WMT ZH-EN task, RIVAL (Qual+Quant) achieved a BLEU score of 33.42, surpassing the GPT-4o baseline of 32.4.
- Generalization: Unlike SFT, which can cause performance to collapse on unseen languages, RIVAL maintained or improved performance on OOD tasks, suggesting it learns fundamental translation strategies.
Conclusion
RIVAL demonstrates that iterative, adversarial optimization is a powerful tool for overcoming the limitations of standard RLHF in machine translation. By keeping the Reward Model "in the loop" and balancing qualitative preferences with quantitative metrics, the framework produces translations that are both semantically creative and lexically accurate. Future work will explore extending this iterative approach to broader generative tasks such as coding and creative writing.
Summary updated