Abstract
Direct preference optimization (DPO) is a successful fine-tuning strategy for aligning large language models with human preferences without the need to train a reward model or employ reinforcement learning. DPO, as originally formulated, relies on binary preference data and fine-tunes a language model to increase the likelihood of a preferred response over a dispreferred response. However, not all preference pairs are equal. Sometimes, the preferred response is only slightly better than the dispreferred one. In other cases, the preference is much stronger. In this paper, we propose a generalization of DPO, termed DPO with an offset (ODPO), that does not treat every preference pair equally during fine-tuning. Intuitively, ODPO requires the difference between the likelihood of the preferred and dispreferred response to be greater than an offset value. The offset is determined based on the extent to which one response is preferred over another. Our experiments on various tasks suggest that ODPO significantly outperforms DPO in aligning language models, especially when the number of preference pairs is limited.
Introduction
Aligning Large Language Models (LLMs) with human values is a critical phase in AI development. While Reinforcement Learning from Human Feedback (RLHF) was the initial standard, its complexity and computational expense led to the development of Direct Preference Optimization (DPO). DPO offers a simpler, more stable alternative by bypassing reward model training entirely. However, DPO treats all preference pairs as binary choices, ignoring the magnitude of preference. Whether a human "slightly prefers" response A over B, or finds A significantly better because B is toxic, DPO treats the gradient update the same. This paper introduces Direct Preference Optimization with an Offset (ODPO) to address this limitation by incorporating the strength of human preferences into the optimization process.
Methodology: Introducing the Offset
Standard DPO assumes human preferences follow the Bradley-Terry model, which focuses solely on the probability of one response being ranked higher than another, ignoring the magnitude of that preference. ODPO generalizes the DPO framework by introducing a dynamic "offset" ($\Delta_r$) into the loss function. This offset forces the model to prioritize the margin between preferred ($y_w$) and dispreferred ($y_l$) responses based on the strength of the preference.
The ODPO loss function is defined as: $$\mathcal{L}_{ODPO}(\theta) = -\mathbb{E}[\log \sigma(\hat{r}_\theta(x, y_w) - \hat{r}_\theta(x, y_l) - \Delta_r)]$$
Where $\Delta_r$ is the offset. If $\Delta_r = 0$, the framework reverts to standard DPO. When $\Delta_r > 0$, the model is penalized unless the gap between the preferred and dispreferred response is at least as large as the offset. The authors derive this offset using the Gumbel-max trick and find that a log-scaled difference between quality scores (like Likert scales) works best to prevent model instability.
Experimental Evaluation
The researchers evaluated ODPO across three distinct alignment challenges:
- Sentiment Control: Steering GPT2-Large toward positive sentiment using the IMDB movie reviews dataset.
- Toxicity Control: Fine-tuning GPT-neo-2.7b to avoid harmful content using RealToxicityPrompts.
- Summarization: Using the Reddit TL;DR dataset with GPTJ-6B, where human-provided Likert scores (1 to 7) defined the offsets.
Key Results and Insights
ODPO consistently outperformed DPO, particularly in low-data regimes. In sentiment tasks, ODPO demonstrated superior Pareto dominance, achieving higher target rewards (positive sentiment) for the same KL divergence cost compared to the base model.
Data Efficiency and Win Rates
One of the most significant findings is ODPO's data efficiency. When only 5,000 preference pairs were available, ODPO significantly outperformed DPO, suggesting the offset acts as a powerful regularizer. In summarization tasks, ODPO-trained models achieved a 62% win rate over DPO in head-to-head comparisons at a sampling temperature of 0.5. In toxicity control with limited data, DPO failed to meaningfully reduce toxicity, while ODPO cut the toxicity probability nearly in half.
Conclusion
ODPO provides a robust generalization of DPO that leverages preference magnitude to improve alignment. While it requires magnitude information (such as Likert scores or toxicity metrics), its gains in data efficiency and alignment quality make it a compelling choice for modern LLM fine-tuning. Future work may explore automated methods for determining optimal offsets or testing the framework on larger architectures like Llama-3.
Summary updated