When is RL better than DPO in RLHF? A Representation and Optimization Perspective

Ziniu Li, Tian Xu, Yang YuICLR 2024 · OpenReview

Overview

This paper investigates the trade-offs between Reinforcement Learning (RL) and Direct Preference Optimization (DPO) for aligning Large Language Models (LLMs). The authors find that while DPO is simpler to implement, it is highly sensitive to "representation misspecification." In contrast, RL-based methods can achieve superior alignment by decoupling the reward model from the policy and leveraging sufficient online updates.

The Alignment Problem

Aligning LLMs with human preferences is typically done via Reinforcement Learning from Human Feedback (RLHF). Currently, two dominant paradigms exist:

  1. RL-based methods (e.g., PPO, ReMax): These involve training an explicit reward model from preference data and then using RL to maximize that reward.
  2. Direct Preference Optimization (DPO): This newer approach skips the explicit reward model, instead optimizing the policy directly using a static dataset of preferences.

While DPO has gained massive popularity due to its stability and simplicity, it remains unclear if it is fundamentally "better" than RL. This paper seeks to answer: Under what mathematical conditions does RL outperform DPO?

Analytical Framework

The authors analyze the problem through the lens of contextual bandits, a standard mathematical framework for alignment. They compare the two approaches based on two distinct perspectives:

  • Representation Perspective: They examine what happens when the feature representation of the policy ($\phi_\pi$) does not match the feature representation of the true reward ($\phi_r$). In DPO, the reward is implicitly tied to the policy's log-probabilities. If the policy's architecture is "misspecified" (i.e., it cannot perfectly represent the reward features), DPO may fail to align correctly.
  • Optimization Perspective: They derive error bounds for RL-based methods, focusing on the impact of "online updates." RL methods generate new responses from the current policy during training, whereas DPO typically relies on a static, offline dataset.

To allow for precise theoretical control and ground-truth comparisons, the study primarily uses a synthetic linear bandit task. The datasets include a preference dataset of 20 samples to train the initial reward/policy, a preference-free dataset of 100 prompts used by the RL agent to estimate state distributions, and a test dataset of 100 preference pairs to evaluate classification accuracy.

Key Insights

  • The DPO Representation Trap: Because DPO reparameterizes the reward using the language model itself, the "quality" of its reward is strictly limited by the representation power of the policy network. If the reward requires a different architecture than the policy (e.g., a reward model that looks at joint state-action features vs. a policy that focuses on state features), DPO suffers.
  • RL's Architectural Flexibility: RL-based methods learn a separate reward function. This allows the reward model to have a different architecture or be trained on different data than the policy, making it more robust to misspecification.
  • The "Online" Advantage: RL's ability to sample new responses from its own evolving policy (online optimization) allows it to explore the distribution more effectively than DPO's static approach.
  • Data Efficiency: RL can use "cheap" unlabeled prompts to improve its performance, whereas DPO is often restricted to the labeled preference dataset.

Empirical Results

The empirical findings strongly support the theoretical analysis, particularly regarding representation misspecification:

  • Preference Accuracy: When representations are perfectly matched ($\phi_\pi = \phi_r$), DPO and RL perform similarly. However, when they are misspecified ($\phi_\pi \neq \phi_r$), DPO’s accuracy lags significantly behind RL.
Preference Classification Accuracy of DPO vs. RL
  • Optimality Gap: In the misspecified scenario, RL achieves a much lower "optimality gap" (a measure of how far the policy is from the theoretical best) compared to DPO.
Optimality Gap of DPO vs. RL
  • The "Sufficient Update" Threshold: The authors demonstrate that RL's superiority is contingent on having enough samples. If the number of prompts or responses used to estimate gradients is too low, the optimality gap for RL increases (from ~0.28 to ~0.30), though it often still outperforms DPO in difficult scenarios.

Limitations and Future Work

While mathematically rigorous, the study uses linear contextual bandits. Real-world LLMs use complex non-linear transformers and have massive action spaces (vocabularies), which may introduce additional dynamics not captured here. Furthermore, the primary challenge for RL remains the computational overhead of the "online" loop (generating responses during training), which DPO avoids.

Future directions include exploring more complex architectures and investigating how these theoretical findings translate to large-scale benchmarks like MT-Bench or AlpacaEval.

Summary updated

Abstract

Aligning large language models with human preferences is important, and there are two kinds of alignment methods. The first class of algorithms is based on reinforcement learning (RL), which involves learning a reward function from a human preference dataset and improving performance via online reward maximization. Another class is characterized by direct preference optimization, exemplified by DPO (Rafailov et al., 2023), which learns an implicit reward and improves performance directly using a static offline dataset. Which algorithm performs well? We investigate this question using contextual bandits, which serve as mathematical models for alignment. We have two findings: First, we show that DPO may suffer from a reward quality issue when the feature representation is misspecified. Second, we present the error bounds for RL algorithms and show that they achieve the best improvement when the online updates are sufficient. The code to reproduce our results is available at https://github.com/liziniu/policy_optimization.
BibTeX
@inproceedings{li2024when,
  author = {Ziniu Li and Tian Xu and Yang Yu},
  title = {{When is RL better than DPO in RLHF? A Representation and Optimization Perspective}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=lNEFatlsQb},
}
Figure