BPO: Revisiting Preference Modeling in Direct Preference Optimization

Lin Sun, Chuang Liu, Peng Liu, Bingyang Li, Weijia Lu, Ning Wu

Abstract

Direct Preference Optimization (DPO) has emerged as a popular method for aligning Large Language Models (LLMs) with human preferences. While DPO effectively preserves the relative ordering between chosen and rejected responses through pairwise ranking losses, it often neglects absolute reward magnitudes. This oversight can decrease the likelihood of chosen responses and increase the risk of generating out-of-distribution responses, leading to poor performance. We term this issue Degraded Chosen Responses (DCR). To address this, we propose Balanced Preference Optimization (BPO), a novel framework that dynamically balances the optimization of chosen and rejected responses through two key components: a balanced reward margin and a gap adaptor. Experimental results on multiple mathematical reasoning tasks show that BPO significantly outperforms DPO, improving accuracy by +10.1% with Llama-3.1-8B-Instruct and +11.7% with Qwen2.5-Math-7B. Remarkably, our algorithm requires only a single line of code modification, making it simple to implement and fully compatible with existing DPO-based frameworks.

The Problem: Degraded Chosen Responses (DCR)

Direct Preference Optimization (DPO) is the standard for aligning LLMs by teaching them to prefer a "chosen" response over a "rejected" one. However, DPO possesses a hidden pathology: it focuses solely on the relative difference between the two. Because it is mathematically easier for a model to decrease the probability of a specific bad response than to increase the probability of a specific good one, DPO often ends up lowering the likelihood of both.

As long as the rejected response's probability drops faster, the loss function is satisfied. This "race to the bottom" pushes the model toward generating out-of-distribution (OOD) responses, which severely degrades performance on complex reasoning tasks like mathematics.

Training dynamics showing how standard DPO (left) leads to negative rewards for chosen responses, while BPO (middle/right) maintains positive chosen rewards.

Balanced Preference Optimization (BPO)

BPO addresses the DCR issue by replacing the standard DPO margin ($\rho = r_w - r_l$) with a Balanced Reward Margin:

$$\rho = \min(r_w, -\alpha r_l)$$

This formulation introduces two critical mechanisms:

  1. The Min Operator: By taking the minimum, the model is forced to focus on the "weaker" component. If the reward for the chosen answer ($r_w$) is low, the model prioritizes raising it. If the rejected answer is too likely, it focuses on suppression.
  2. The Gap Adaptor ($\alpha$): This hyperparameter tunes the penalty on rejected responses. A smaller $\alpha$ prevents the model from over-optimizing against specific negative examples, maintaining stability.

BPO is a drop-in replacement requiring only a single line of code change in existing DPO frameworks, without needing additional constraints or complex loss terms.

Experimental Results

We evaluated BPO using the Open Reasoner Zero 57k dataset across five competition-level math benchmarks: AIME2024, MATH500, AMC2023, MinervaMath, and OlympiadBench.

Comparison of Accuracy (%) between DPO and BPO across multiple benchmarks for Llama-3.1-8B and Qwen2.5-Math-7B models.

BPO demonstrated substantial gains over standard DPO and its variants:

  • Llama-3.1-8B-Instruct: Average accuracy jumped from 18.8% to 28.9% (+10.1%).
  • Qwen2.5-Math-7B: Average accuracy rose from 35.0% to 46.7% (+11.7%).
  • AIME2024 Performance: On the notoriously difficult AIME2024 benchmark, BPO achieved 30.0% accuracy with Qwen2.5-Math-7B, a massive leap from the 6.7% baseline.
  • Comparison with Variants: BPO outperformed other DPO improvements, including IPO (+3.6% gain), SLiC (+5.0% gain), and Cal-DPO (+3.1% gain).

Key Insights

  • Addressing Task Difficulty: Increasing the probability of a specific correct answer is fundamentally harder than decreasing the probability of a wrong one. BPO accounts for this asymmetry.
  • Stability through Balance: By ensuring $r_w$ remains positive and continues to increase, BPO prevents the "likelihood collapse" seen in standard DPO.
  • Efficiency: Because BPO's gradient only updates one side of the pair (chosen or rejected) depending on which is weaker, it effectively reduces computational overhead during gradient calculation.

Conclusion and Future Work

BPO provides a simple yet powerful solution to the DCR issue in offline preference optimization. While currently focused on offline methods, future work will explore applying the balanced reward margin to on-policy learning, where the model generates new responses and receives real-time feedback. This could further unlock the reasoning potential of aligned LLMs.

Summary updated

Abstract

Direct Preference Optimization (DPO) have emerged as a popular method for aligning Large Language Models (LLMs) with human preferences. While DPO effectively preserves the relative ordering between chosen and rejected responses through pairwise ranking losses, it often neglects absolute reward magnitudes. This oversight can decrease the likelihood of chosen responses and increase the risk of generating out-of-distribution responses, leading to poor performance. We term this issue Degraded Chosen Responses (DCR).To address this issue, we propose Balanced Preference Optimization (BPO), a novel framework that dynamically balances the optimization of chosen and rejected responses through two key components: balanced reward margin and gap adaptor. Unlike previous methods, BPO can fundamentally resolve DPO's DCR issue, without introducing additional constraints to the loss function. Experimental results on multiple mathematical reasoning tasks show that BPO significantly outperforms DPO, improving accuracy by +10.1% with Llama-3.1-8B-Instruct (18.8% to 28.9%) and +11.7% with Qwen2.5-Math-7B (35.0% to 46.7%). It also surpasses DPO variants by +3.6% over IPO (43.1%), +5.0% over SLiC (41.7%), and +3.1% over Cal-DPO (43.6%) on the same model. Remarkably, our algorithm requires only a single line of code modification, making it simple to implement and fully compatible with existing DPO-based frameworks.
BibTeX
@misc{sun2025bpo,
  author = {Lin Sun and Chuang Liu and Peng Liu and Bingyang Li and Weijia Lu and Ning Wu},
  title = {{BPO: Revisiting Preference Modeling in Direct Preference Optimization}},
  year = {2025},
  eprint = {2506.03557},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.03557},
}
Figure