Dataset Reset Policy Optimization for RLHF

Jonathan D. Chang, Wenhao Zhan, Owen Oertell, Kianté Brantley, Dipendra Misra, Jason D. Lee, Wen SunarXiv 2024 · DOI · OpenReview

Introduction

Reinforcement Learning from Human Feedback (RLHF) has become the standard methodology for aligning Large Language Models (LLMs) with human preferences. The traditional pipeline involves two distinct phases: learning a reward model from an offline preference dataset and optimizing a policy using online reinforcement learning (typically PPO) to maximize that reward. However, this approach often suffers from exploration inefficiencies, as the model must generate entire sequences from scratch to receive feedback, frequently missing high-reward regions of the state space.

The Exploration Challenge

In standard RLHF, the model starts generating from the beginning of a prompt. To receive a high reward, it must "stumble" upon a sequence of tokens that the reward model favors. This is a difficult search problem. Meanwhile, the offline datasets used to train reward models already contain "informative states"—prefixes of sequences that lead to high-quality outcomes. Dataset Reset Policy Optimization (DR-PO) is designed to exploit these existing states to jump-start the online exploration process.

Methodology: Dataset Reset Policy Optimization

DR-PO introduces a Dataset Reset mechanism that leverages the unique properties of generative models. In LLMs, the "state" is the sequence of tokens generated so far. Because transitions are deterministic (appending a token), the model can be "reset" to any intermediate state simply by providing it with a specific prefix from the offline dataset.

The DR-PO Algorithm

  1. Reward Model Training: A reward model $\hat{r}$ is fitted to an offline preference dataset using Maximum Likelihood Estimation (MLE).
  2. Reset-Based Data Collection: During the online phase, the algorithm samples a trajectory from the offline dataset and selects a random intermediate prefix. The policy is "reset" to this prefix, and the model generates the completion from that point forward.
  3. Policy Optimization: The trajectories collected via these resets are used to update the policy. The authors utilize PPO in practice, while providing theoretical analysis via Natural Policy Gradient.
  4. KL Regularization: To prevent "reward hacking" and maintain the model's natural language capabilities, a KL-divergence penalty is applied relative to the initial Supervised Fine-Tuned (SFT) model.

Theoretical Foundations

DR-PO is supported by a theoretical framework showing that it can learn a policy at least as good as any policy covered by the offline dataset. This provides a more robust guarantee than many purely empirical RLHF methods, particularly under general function approximation. By starting from human-preferred prefixes, the model spends more time exploring high-value regions of the state space rather than wasting computational budget on nonsensical or low-reward beginnings.

Experimental Results

The authors evaluated DR-PO across Pythia models ranging from 125M to 6.9B parameters on several benchmarks, including TL;DR Summarization and the Anthropic Helpful Harmful (HH) dataset.

Key Findings

  • Superior Win-Rates: On the TL;DR task, DR-PO achieved a 70.2% win-rate against human references (judged by GPT-4), significantly outperforming PPO (62.3%) and DPO (52.6%).
  • Optimized Pareto Frontier: DR-PO achieves a better tradeoff between reward maximization and KL divergence than standard PPO, reaching higher reward scores while staying closer to the reference model distribution.
  • Robust Generalization: In zero-shot transfer tests on the CNN/DailyMail dataset, DR-PO maintained its performance advantage. This suggests the model learns general principles of quality rather than merely memorizing the training data.
  • Scalability: The performance gains remained consistent as model size increased, confirming the method's viability for large-scale LLM alignment.

Conclusion

DR-PO addresses a fundamental inefficiency in RLHF by bridging the gap between offline data and online exploration. By treating the offline dataset as a source of informative "reset" points, the algorithm achieves higher performance and better theoretical guarantees than existing baselines. Future work may explore the application of dataset resets in other generative domains, such as diffusion models for image generation, where deterministic reverse processes make similar resets feasible.

Summary updated

Abstract

Reinforcement Learning (RL) from Human Preference-based feedback is a popular paradigm for fine-tuning generative models, which has produced impressive models such as GPT-4 and Claude3 Opus. This framework often consists of two steps: learning a reward model from an offline preference dataset followed by running online RL to optimize the learned reward model. In this work, leveraging the idea of reset, we propose a new RLHF algorithm with provable guarantees. Motivated by the fact that offline preference dataset provides informative states (i.e., data that is preferred by the labelers), our new algorithm, Dataset Reset Policy Optimization (DR-PO), integrates the existing offline preference dataset into the online policy training procedure via dataset reset: it directly resets the policy optimizer to the states in the offline dataset, instead of always starting from the initial state distribution. In theory, we show that DR-PO learns to perform at least as good as any policy that is covered by the offline dataset under general function approximation with finite sample complexity. In experiments, we demonstrate that on both the TL;DR summarization and the Anthropic Helpful Harmful (HH) dataset, the generation from DR-PO is better than that from Proximal Policy Optimization (PPO) and Direction Preference Optimization (DPO), under the metric of GPT4 win-rate. Code for this work can be found at https://github.com/Cornell-RL/drpo.
BibTeX
@misc{chang2024dataset,
  author = {Jonathan D. Chang and Wenhao Zhan and Owen Oertell and Kianté Brantley and Dipendra Misra and Jason D. Lee and Wen Sun},
  title = {{Dataset Reset Policy Optimization for RLHF}},
  year = {2024},
  eprint = {2404.08495},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.08495},
}
Figure