Abstract
Direct Preference Optimization (DPO) has emerged as a powerful alternative to traditional Reinforcement Learning from Human Feedback (RLHF) for aligning large language models (LLMs). By bypassing the need for an explicit reward model, DPO simplifies the fine-tuning process. However, standard DPO treats all preference pairs as binary, ignoring the relative intensity of human preferences. This paper introduces Direct Preference Optimization with an Offset (ODPO), a generalization of DPO that incorporates a margin, or offset, into the optimization objective. This offset reflects the magnitude of preference between two responses. Our results across sentiment control, toxicity reduction, and summarization tasks demonstrate that ODPO significantly outperforms standard DPO, particularly in data-constrained environments.
Introduction
Aligning LLMs with human intent is a critical step in model development. While RLHF is the industry standard, it is computationally expensive and notoriously difficult to stabilize due to the complexities of training a separate reward model and utilizing Proximal Policy Optimization (PPO). DPO revolutionized this space by showing that the optimal policy can be derived directly from the preference data, effectively treating the LLM as its own reward model.
Despite its success, DPO possesses a fundamental limitation: it treats every preference pair as an equal binary signal. In practice, human preferences are rarely uniform. A choice between a helpful response and a toxic one represents a much stronger preference than a choice between two helpful responses with slight stylistic differences. By ignoring these nuances, DPO discards valuable information. ODPO addresses this by requiring the model to favor a preferred response over a dispreferred one by a specific margin—the offset—proportional to the intensity of the preference.
The ODPO Framework
Mathematical Formulation
The core innovation of ODPO is the modification of the DPO loss function. In standard DPO, the model minimizes a loss based on the probability that the reward of the winning response ($\hat{r}_w$) exceeds the reward of the losing response ($\hat{r}_l$). ODPO introduces an offset $\Delta_r$:
Standard DPO Loss: $\log \sigma(\hat{r}_w - \hat{r}_l)$ ODPO Loss: $\log \sigma(\hat{r}_w - \hat{r}_l - \Delta_r)$
This formulation is theoretically grounded in the Bradley-Terry model. By assuming the latent rewards are subject to Gumbel-distributed noise, the authors prove that the probability of one reward exceeding another by a specific margin follows a sigmoid distribution shifted by that margin. This ensures that ODPO is not merely a heuristic but a mathematically sound extension of preference learning.
Defining the Offset
The offset $\Delta_r$ can be derived from various sources of "intensity" data:
- Point-wise Scores: Utilizing external classifiers to determine toxicity or sentiment scores.
- Likert Scales: Using human-annotated scales (e.g., 1–7) to quantify the gap between responses.
- Reward Log-Differences: Leveraging existing reward model scores where available.
Experimental Setup
The researchers evaluated ODPO against standard DPO across three distinct domains:
- Sentiment Control: Fine-tuning GPT-2 Large on the IMDB dataset to generate positive movie reviews.
- Toxicity Reduction: Utilizing REALTOXICITYPROMPTS to steer GPT-Neo-2.7B away from harmful content.
- Abstractive Summarization: Fine-tuning GPT-J-6B on the Reddit TL;DR dataset, which includes human quality scores on a 7-point Likert scale.
Key Insights and Results
Superior Pareto Frontiers
In alignment tasks, there is an inherent trade-off between maximizing the reward (alignment) and minimizing the KL divergence from the base model (preserving general capabilities). ODPO consistently achieves a superior Pareto frontier compared to DPO. This means that for any given level of model drift, ODPO achieves a higher reward, effectively reducing the risk of "reward hacking."
Robustness in Low-Data Regimes
One of the most significant findings is ODPO's efficiency when training data is scarce. In the toxicity control task using only 8,000 preference pairs, standard DPO struggled to produce a meaningful shift in model behavior. In contrast, ODPO successfully halved the toxicity of generated outputs while maintaining a low KL divergence. This suggests that the additional "signal" provided by the preference intensity allows the model to learn the underlying reward landscape more rapidly.
Human and Model Evaluation
When evaluated by GPT-4—which serves as a proxy for human judgment—ODPO-trained models demonstrated a clear advantage. In the summarization task, ODPO achieved a 62% win rate over standard DPO at a sampling temperature of 0.5. The authors also found that scaling the reward difference with a log function provided the most stable performance, allowing for high rewards without aggressive divergence from the reference supervised fine-tuning (SFT) model.
Conclusion and Future Work
ODPO provides a simple yet effective generalization of Direct Preference Optimization that leverages the nuanced intensity of human feedback. By moving beyond binary labels, ODPO extracts more information from each preference pair, leading to better alignment and greater data efficiency.
While ODPO requires datasets with intensity information (such as scores or scales), its performance gains suggest that collecting such metadata is highly valuable. Future research may focus on automated methods for determining optimal offset scaling and applying ODPO to multi-objective alignment scenarios where models must balance competing goals like safety, honesty, and helpfulness.
Summary updated