Abstract
Reinforcement Learning from Human Feedback (RLHF) typically involves a trade-off between maximizing human-defined rewards and maintaining proximity to a reference language model. We reframe RLHF as a bi-objective Pareto optimization problem and identify a necessary condition for simultaneous improvement in both objectives. Based on this, we introduce Reward Dropout, a simple filtering technique that ignores low-reward samples during training. Our experiments across various text style transfer benchmarks demonstrate that focusing on a small fraction of high-reward samples consistently leads to Pareto improvements, even when training from scratch.
Introduction
In standard RLHF, models are tasked with two conflicting goals: generating high-reward text (alignment) and maintaining a high likelihood relative to a "safe" reference model. Pushing for higher rewards often causes the model's likelihood to crash, a phenomenon known as "reward hacking" or "bi-objective degeneration," where the model fails to optimize either objective effectively.
This work addresses the lack of theoretical understanding regarding when RLHF successfully improves both objectives. By treating RLHF as a bi-objective optimization problem, we can define a specific condition for "Pareto improvement"—where at least one objective is improved without degrading the other.
Methods: The Pareto Condition and Reward Dropout
We derive a first-order optimality condition for the standard RLHF objective, identifying a necessary condition for Pareto improvement: the "bi-objective reward" (the sum of the reward and the log-likelihood of the behavior model) must be positive.
To enforce this condition, we propose Reward Dropout. The core algorithm involves:
- Calculating the bi-objective reward for each sample in a training batch.
- Filtering samples: Setting the rewards of samples that fall below a certain threshold $\delta$ to zero.
- Quantile Dropout: In practice, we sort samples by reward and drop the bottom $\gamma$-percent (e.g., the bottom 80% or 95%).
By focusing updates only on high-reward samples that satisfy the Pareto condition, the gradient updates are theoretically guaranteed to move the model toward a region where both likelihood and reward increase.
Experimental Setup
We evaluated Reward Dropout using two common text style transfer benchmarks:
- Yelp Sentiment Dataset: Focused on positive and negative sentiment attributes.
- AG_News Topic Dataset: Focused on World, Sport, Business, and Sci/Tech attributes.
- Anthropic Harmlessness Dataset: Used to test the method on implicit safety attributes.
We utilized OPT-6.7B as the reference model and tested various target models, including GPT-2 (124M, 774M), XGLM-1.7B, and OPT-6.7B.
Key Results and Insights
Consistent Pareto Gains
Across almost all tasks and model sizes, "Quantile Dropout" significantly outperformed standard RLHF and random dropout baselines. It consistently increased the sum of accuracy (likelihood) and reward.
The "Less is More" Phenomenon
Surprisingly, paying attention to only the top 5-10% of samples (high $\gamma$ values like 0.9 or 0.95) led to the greatest Pareto improvements. This suggests that a small number of high-quality samples are more valuable for alignment than a large volume of mediocre ones. Furthermore, Reward Dropout proved highly effective even when the target model was initialized with random parameters, suggesting that traditional pretraining might be skippable for certain task-specific alignment goals.
Scalability and Quality
The method scaled effectively from small models (GPT2-124M) to larger ones (OPT-6.7B). Evaluation using metrics like BLEU and METEOR showed that Reward Dropout improved word-level alignment and stylistic control, effectively mitigating the "curse of recursion" common in on-policy RL.
Limitations and Future Work
While effective for explicit labels like sentiment or topic, the method was less successful on the "Harmlessness" dataset, likely due to the difficulty of reward models in capturing subtle linguistic nuances associated with safety. Additionally, while Reward Dropout guarantees updates in the direction of Pareto improvement, it does not yet guarantee convergence to a globally Pareto-efficient policy. Future research will explore traditional multi-objective optimization algorithms and the application of Reward Dropout to even larger state-of-the-art models.
Summary updated