Abstract
Policy gradient algorithms have found wide adoption in NLP, but have recently become subject to criticism, doubting their suitability for NMT. Choshen et al. (2020) identify multiple weaknesses and suspect that their success is determined by the shape of output distributions rather than the reward. In this paper, we revisit these claims and study them under a wider range of configurations. Our experiments on in-domain and cross-domain adaptation reveal the importance of exploration and reward scaling, and provide empirical counter-evidence to these claims.
Introduction
Standard Neural Machine Translation (NMT) models are typically trained using Maximum Likelihood Estimation (MLE), which optimizes for token-level accuracy. However, translation quality is usually evaluated using sequence-level metrics like BLEU. Reinforcement Learning (RL) offers a way to bridge this gap by optimizing the model directly on these metrics.
Recently, a high-profile critique by Choshen et al. (2020) argued that RL's success in NMT was largely an illusion. They claimed that the reward signal (e.g., BLEU score) didn't actually drive learning; instead, models improved simply because RL training made output distributions "peakier"—essentially reinforcing the model's existing biases. If true, RL would be ineffective for incorporating real-world human preferences. This study revisits these claims to determine if RL is a "dead end" or if previous critiques overlooked critical implementation details.
Methods
The study evaluates two primary RL algorithms:
- Policy Gradient (PG): The standard REINFORCE algorithm, which samples a translation and scales the gradient by the reward.
- Minimum Risk Training (MRT): A more stable variant that samples multiple translations and optimizes to minimize expected loss across them.
To test the "peakiness" hypothesis, the authors introduced several controls:
- Softmax Temperature ($\tau$): Adjusting $\tau$ controls exploration. Higher temperatures prevent the model from becoming too "peaky" too quickly.
- Reward Scaling and Baselines: By subtracting an "average reward baseline," the authors enable negative reinforcement, allowing the model to learn what not to do.
- Constant Rewards: Informative rewards (BLEU) were compared against a constant reward of 1.0 to see if improvement occurs without a real signal.
Key Insights: Rewards and Upwards Mobility
A central finding is that rewards actually matter. In cross-domain adaptation (moving from WMT15 to IWSLT14), a "constant reward" baseline failed to improve the model, while informative rewards led to significant gains. This directly contradicts the idea that RL gains are purely structural.
Furthermore, the authors demonstrate "upwards mobility." RL can take tokens initially ranked very low by the model and move them into the top ranks. This proves the model isn't just "stuck" with its pre-trained biases.
The study also highlights the importance of negative rewards. Without reward scaling (baselines), the model's ability to learn is severely limited. Finally, RL-trained models appear more robust to the "beam search curse," where performance typically degrades with larger beam sizes.
Experimental Results
The most compelling evidence comes from cross-domain performance. When adapting a WMT-trained model to the IWSLT domain, MRT improved the BLEU score significantly, moving from 20.35 to 27.08.
As shown in the table above, Policy Gradient with informative rewards reached 24.53 BLEU ($k=5$), whereas PG with a constant reward stayed flat at 20.35. Additionally, by increasing the softmax temperature ($\tau = 1.2$), the authors achieved performance gains while decreasing the model's peakiness, proving that peakiness is a byproduct, not the cause, of improvement.
Conclusion
While RL methods still depend on strong MLE pre-training due to the vast search space of NMT, this research provides strong empirical evidence that RL effectively utilizes reward signals. Future work should explore how these findings translate to noisier, real-world human feedback and address the computational costs associated with stable methods like MRT.
Summary updated