Introduction
Reinforcement Learning from Human Feedback (RLHF) has become the standard methodology for aligning Large Language Models (LLMs) with human preferences. The traditional pipeline involves two distinct phases: learning a reward model from an offline preference dataset and optimizing a policy using online reinforcement learning (typically PPO) to maximize that reward. However, this approach often suffers from exploration inefficiencies, as the model must generate entire sequences from scratch to receive feedback, frequently missing high-reward regions of the state space.
The Exploration Challenge
In standard RLHF, the model starts generating from the beginning of a prompt. To receive a high reward, it must "stumble" upon a sequence of tokens that the reward model favors. This is a difficult search problem. Meanwhile, the offline datasets used to train reward models already contain "informative states"—prefixes of sequences that lead to high-quality outcomes. Dataset Reset Policy Optimization (DR-PO) is designed to exploit these existing states to jump-start the online exploration process.
Methodology: Dataset Reset Policy Optimization
DR-PO introduces a Dataset Reset mechanism that leverages the unique properties of generative models. In LLMs, the "state" is the sequence of tokens generated so far. Because transitions are deterministic (appending a token), the model can be "reset" to any intermediate state simply by providing it with a specific prefix from the offline dataset.
The DR-PO Algorithm
- Reward Model Training: A reward model $\hat{r}$ is fitted to an offline preference dataset using Maximum Likelihood Estimation (MLE).
- Reset-Based Data Collection: During the online phase, the algorithm samples a trajectory from the offline dataset and selects a random intermediate prefix. The policy is "reset" to this prefix, and the model generates the completion from that point forward.
- Policy Optimization: The trajectories collected via these resets are used to update the policy. The authors utilize PPO in practice, while providing theoretical analysis via Natural Policy Gradient.
- KL Regularization: To prevent "reward hacking" and maintain the model's natural language capabilities, a KL-divergence penalty is applied relative to the initial Supervised Fine-Tuned (SFT) model.
Theoretical Foundations
DR-PO is supported by a theoretical framework showing that it can learn a policy at least as good as any policy covered by the offline dataset. This provides a more robust guarantee than many purely empirical RLHF methods, particularly under general function approximation. By starting from human-preferred prefixes, the model spends more time exploring high-value regions of the state space rather than wasting computational budget on nonsensical or low-reward beginnings.
Experimental Results
The authors evaluated DR-PO across Pythia models ranging from 125M to 6.9B parameters on several benchmarks, including TL;DR Summarization and the Anthropic Helpful Harmful (HH) dataset.
Key Findings
- Superior Win-Rates: On the TL;DR task, DR-PO achieved a 70.2% win-rate against human references (judged by GPT-4), significantly outperforming PPO (62.3%) and DPO (52.6%).
- Optimized Pareto Frontier: DR-PO achieves a better tradeoff between reward maximization and KL divergence than standard PPO, reaching higher reward scores while staying closer to the reference model distribution.
- Robust Generalization: In zero-shot transfer tests on the CNN/DailyMail dataset, DR-PO maintained its performance advantage. This suggests the model learns general principles of quality rather than merely memorizing the training data.
- Scalability: The performance gains remained consistent as model size increased, confirming the method's viability for large-scale LLM alignment.
Conclusion
DR-PO addresses a fundamental inefficiency in RLHF by bridging the gap between offline data and online exploration. By treating the offline dataset as a source of informative "reset" points, the algorithm achieves higher performance and better theoretical guarantees than existing baselines. Future work may explore the application of dataset resets in other generative domains, such as diffusion models for image generation, where deterministic reverse processes make similar resets feasible.
Summary updated