Abstract
Aligning intelligent agents with human preferences and values is important. This paper examines two popular alignment methods: Direct Preference Optimization (DPO) and Reward-Model-Based Policy Optimization (RMB-PO). A variant of RMB-PO, referred to as RMB-PO+, is also considered. These methods, either explicitly or implicitly, learn a reward model from preference data and differ in the data used for policy optimization to unlock the generalization ability of the reward model. In particular, compared with DPO, RMB-PO additionally uses policy-generated data, and RMB-PO+ further leverages new, preference-free data. Our study, conducted through controlled and synthetic experiments, demonstrates that DPO performs poorly, whereas RMB-PO+ performs the best. In particular, even when providing the policy model with a good feature representation, we find that policy optimization with adequate out-of-preference data significantly improves performance by harnessing the reward model's generalization capabilities.
Introduction
In the rapidly evolving landscape of Large Language Model (LLM) alignment, two primary paradigms have emerged: Direct Preference Optimization (DPO) and Reinforcement Learning from Human Feedback (RLHF) using Reward-Model-Based approaches (e.g., PPO). While DPO has gained significant traction due to its simplicity—eliminating the need to maintain and train a separate reward model—recent research suggests that this simplicity may come at the cost of performance and robustness.
The core challenge in alignment is ensuring that AI agents adhere to human preferences across a wide variety of scenarios. This paper investigates whether the ability to leverage "out-of-preference" data—data points not explicitly labeled in the initial human preference dataset—provides a decisive advantage for reward-model-based methods over reward-model-free alternatives like DPO.
Methodology: RMF vs. RMB Approaches
The researchers categorize alignment strategies into three distinct classes based on their data utilization:
- Reward-Model-Free (RMF-PO): Represented by DPO, this approach treats alignment as a classification problem. It optimizes the policy using only the static pairs provided in the preference dataset, effectively performing behavioral cloning on preferences.
- Reward-Model-Based (RMB-PO): This is the standard RLHF approach. It first trains a reward model from preferences and then optimizes the policy by evaluating new responses generated by the model itself against the learned reward model.
- RMB-PO+: An extension of RMB-PO that incorporates "preference-free" data—unlabeled prompts. The model generates responses to these new prompts, and the reward model provides pseudo-labels for optimization.
The critical distinction lies in the use of "out-of-preference" data. While DPO is restricted to the human-labeled distribution, RMB-PO methods can explore the reward landscape by evaluating the policy's current outputs, even in regions the original preference data did not cover.
Experimental Insights
To quantify the performance gap, the authors utilized synthetic Contextual Bandit environments (Linear and Neural). These controlled settings allow for the calculation of an "optimality gap"—the difference between a method's performance and the theoretical maximum.
The Generalization Advantage
A key finding is that a learned reward model often generalizes more effectively than a policy trained directly on preference pairs. By using the reward model to label new, unseen data, the system "distills" this superior generalization into the policy. DPO, by contrast, suffers from "blind spots"; if the policy generates responses outside the narrow distribution of the training set, DPO lacks the signal to correct it.
Performance Results
In complex Neural Bandit tasks, which mimic the non-linear nature of real-world LLMs, the superiority of reward-model-based approaches becomes evident. RMB-PO+ consistently achieved the lowest optimality gap by leveraging additional unlabeled prompts to better map the reward landscape.
Discussion and Conclusion
The research highlights a fundamental trade-off: while DPO offers computational efficiency and ease of implementation, it may be inherently limited by its inability to utilize out-of-preference data. Reward-model-based approaches, particularly RMB-PO+, demonstrate that "prompts are cheap, but preferences are expensive." By utilizing large volumes of unlabeled data (such as the lmsys-chat-1m dataset) and a robust reward model, developers can achieve significantly better alignment.
Future work remains to address the computational overhead of RMB-PO and the risks of reward hacking as policies explore further out-of-distribution. However, the results suggest that the path to robust, generalized alignment involves embracing the complexity of reward models and the vast potential of unlabeled data.
Summary updated