Direct Preference Optimization with an Offset

Afra Amini, Tim Vieira, Ryan Cotterell

Abstract

Direct Preference Optimization (DPO) has emerged as a powerful alternative to traditional Reinforcement Learning from Human Feedback (RLHF) for aligning large language models (LLMs). By bypassing the need for an explicit reward model, DPO simplifies the fine-tuning process. However, standard DPO treats all preference pairs as binary, ignoring the relative intensity of human preferences. This paper introduces Direct Preference Optimization with an Offset (ODPO), a generalization of DPO that incorporates a margin, or offset, into the optimization objective. This offset reflects the magnitude of preference between two responses. Our results across sentiment control, toxicity reduction, and summarization tasks demonstrate that ODPO significantly outperforms standard DPO, particularly in data-constrained environments.

Introduction

Aligning LLMs with human intent is a critical step in model development. While RLHF is the industry standard, it is computationally expensive and notoriously difficult to stabilize due to the complexities of training a separate reward model and utilizing Proximal Policy Optimization (PPO). DPO revolutionized this space by showing that the optimal policy can be derived directly from the preference data, effectively treating the LLM as its own reward model.

Despite its success, DPO possesses a fundamental limitation: it treats every preference pair as an equal binary signal. In practice, human preferences are rarely uniform. A choice between a helpful response and a toxic one represents a much stronger preference than a choice between two helpful responses with slight stylistic differences. By ignoring these nuances, DPO discards valuable information. ODPO addresses this by requiring the model to favor a preferred response over a dispreferred one by a specific margin—the offset—proportional to the intensity of the preference.

The ODPO Framework

Mathematical Formulation

The core innovation of ODPO is the modification of the DPO loss function. In standard DPO, the model minimizes a loss based on the probability that the reward of the winning response ($\hat{r}_w$) exceeds the reward of the losing response ($\hat{r}_l$). ODPO introduces an offset $\Delta_r$:

Standard DPO Loss: $\log \sigma(\hat{r}_w - \hat{r}_l)$ ODPO Loss: $\log \sigma(\hat{r}_w - \hat{r}_l - \Delta_r)$

This formulation is theoretically grounded in the Bradley-Terry model. By assuming the latent rewards are subject to Gumbel-distributed noise, the authors prove that the probability of one reward exceeding another by a specific margin follows a sigmoid distribution shifted by that margin. This ensures that ODPO is not merely a heuristic but a mathematically sound extension of preference learning.

Defining the Offset

The offset $\Delta_r$ can be derived from various sources of "intensity" data:

  1. Point-wise Scores: Utilizing external classifiers to determine toxicity or sentiment scores.
  2. Likert Scales: Using human-annotated scales (e.g., 1–7) to quantify the gap between responses.
  3. Reward Log-Differences: Leveraging existing reward model scores where available.

Experimental Setup

The researchers evaluated ODPO against standard DPO across three distinct domains:

  • Sentiment Control: Fine-tuning GPT-2 Large on the IMDB dataset to generate positive movie reviews.
  • Toxicity Reduction: Utilizing REALTOXICITYPROMPTS to steer GPT-Neo-2.7B away from harmful content.
  • Abstractive Summarization: Fine-tuning GPT-J-6B on the Reddit TL;DR dataset, which includes human quality scores on a 7-point Likert scale.

Key Insights and Results

Superior Pareto Frontiers

In alignment tasks, there is an inherent trade-off between maximizing the reward (alignment) and minimizing the KL divergence from the base model (preserving general capabilities). ODPO consistently achieves a superior Pareto frontier compared to DPO. This means that for any given level of model drift, ODPO achieves a higher reward, effectively reducing the risk of "reward hacking."

Robustness in Low-Data Regimes

One of the most significant findings is ODPO's efficiency when training data is scarce. In the toxicity control task using only 8,000 preference pairs, standard DPO struggled to produce a meaningful shift in model behavior. In contrast, ODPO successfully halved the toxicity of generated outputs while maintaining a low KL divergence. This suggests that the additional "signal" provided by the preference intensity allows the model to learn the underlying reward landscape more rapidly.

Human and Model Evaluation

When evaluated by GPT-4—which serves as a proxy for human judgment—ODPO-trained models demonstrated a clear advantage. In the summarization task, ODPO achieved a 62% win rate over standard DPO at a sampling temperature of 0.5. The authors also found that scaling the reward difference with a log function provided the most stable performance, allowing for high rewards without aggressive divergence from the reference supervised fine-tuning (SFT) model.

Conclusion and Future Work

ODPO provides a simple yet effective generalization of Direct Preference Optimization that leverages the nuanced intensity of human feedback. By moving beyond binary labels, ODPO extracts more information from each preference pair, leading to better alignment and greater data efficiency.

While ODPO requires datasets with intensity information (such as scores or scales), its performance gains suggest that collecting such metadata is highly valuable. Future research may focus on automated methods for determining optimal offset scaling and applying ODPO to multi-objective alignment scenarios where models must balance competing goals like safety, honesty, and helpfulness.

Summary updated

Abstract

Direct preference optimization (DPO) is a successful fine-tuning strategy for aligning large language models with human preferences without the need to train a reward model or employ reinforcement learning. DPO, as originally formulated, relies on binary preference data and fine-tunes a language model to increase the likelihood of a preferred response over a dispreferred response. However, not all preference pairs are equal. Sometimes, the preferred response is only slightly better than the dispreferred one. In other cases, the preference is much stronger. For instance, if a response contains harmful or toxic content, the annotator will have a strong preference for that response. In this paper, we propose a generalization of DPO, termed DPO with an offset (ODPO), that does not treat every preference pair equally during fine-tuning. Intuitively, ODPO requires the difference between the likelihood of the preferred and dispreferred response to be greater than an offset value. The offset is determined based on the extent to which one response is preferred over another. Our experiments on various tasks suggest that ODPO significantly outperforms DPO in aligning language models, especially when the number of preference pairs is limited.
BibTeX
@inproceedings{amini2024direct,
  author = {Afra Amini and Tim Vieira and Ryan Cotterell},
  title = {{Direct Preference Optimization with an Offset}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=bbzaT3T8Ml},
}
Figure