BPO: Revisiting Preference Modeling in Direct Preference Optimization

Lin Sun, Chuang Liu, Peng Liu, Bingyang Li, Weijia Lu, Ning WuSubmission to ICLR 2026 · Withdrawn

TL;DR

Standard Direct Preference Optimization (DPO) often increases the gap between chosen and rejected responses by driving down the probability of both, a failure mode dubbed Degraded Chosen Responses (DCR). Balanced Preference Optimization (BPO) fixes this with a single line of code by replacing the relative reward margin $(r_w - r_l)$ with a balanced margin $\min(r_w, -\alpha r_l)$. This simple change keeps chosen response likelihoods growing and delivers massive gains in complex mathematical reasoning (+10.1% on Llama-3.1-8B and +11.7% on Qwen2.5-Math-7B).

What Problem Does This Paper Address?

Direct Preference Optimization (DPO) has largely replaced complex RLHF pipelines by directly optimizing language model probabilities using preference pairs (chosen $y_w$ vs. rejected $y_l$). DPO maximizes the relative margin:

$$\rho_\theta = r_w - r_l = \beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}$$

However, direct preference methods suffer from an inherent asymmetry in task difficulty:

  • Lowering the likelihood of an incorrect answer $y_l$ is easy (the model can allocate probability mass to almost any alternative token).
  • Raising the likelihood of a correct, multi-step derivation $y_w$ is hard (the model must pinpoint precise tokens).

As a result, gradient descent takes the path of least resistance: it aggressively pushes down $r_l$ while allowing $r_w$ to drift into negative territory. The authors term this Degraded Chosen Responses (DCR). When absolute probabilities collapse for both chosen and rejected answers, the model inadvertently boosts out-of-distribution (OOD) generation, leading to hallucinations and poor reasoning performance.

Methods

Rather than adding ad-hoc penalty terms or regularizers to the loss function (like DPOP or Cal-DPO), BPO reformulates the reward margin itself:

  1. Balanced Reward Margin: The standard margin is replaced with: $$\rho^b_\theta = \min(r_w, -\alpha r_l)$$ This forces the optimizer to dynamically prioritize whichever component is falling behind. If chosen reward $r_w$ is too low ($r_w < -\alpha r_l$), gradients only update the chosen response. Once chosen responses are sufficiently rewarded, optimization shifts to suppressing rejected responses.

  2. Gap Adaptor ($\alpha$): A hyperparameter $\alpha \in (0, 1]$ tunes the relative weight between reinforcing correct answers and penalizing incorrect ones. Empirical tests show that moderately relaxing the penalty on rejected responses (e.g., $\alpha \approx 0.3 \text{ to } 0.5$) yields optimal performance.

  3. Plug-and-Play Generality: BPO requires changing literally one line of code in existing loss implementations and works across multiple loss families (Bradley-Terry logistic loss, SLiC hinge loss, IPO squared loss, etc.).

Mathematically, BPO guarantees a lower bound on the chosen response probability:

$$\pi_\theta(y_w|x) \ge \exp\left(\frac{\gamma}{\beta}\right) \pi_{\text{ref}}(y_w|x)$$

ensuring that the policy never degrades chosen completions relative to the reference model.

Datasets

  • Training: Trained on Open Reasoner Zero 57k, an open-source mathematical dataset combining subsets from AIME, MATH, Numina-Math, Tulu3 MATH, and OpenR1-Math-220k. Preference pairs were constructed using a max-min outcome reward strategy over 8 sampled trajectories per prompt.
  • Evaluation: Evaluated on five competition-level math benchmarks via pass@1 accuracy:
  • AIME 2024
  • MATH500
  • AMC 2023
  • MinervaMath
  • OlympiadBench

Key Insights and Ideas

  • Gradients update only the weaker side: Unlike DPO, which always backpropagates through both $y_w$ and $y_l$, BPO's $\min(\cdot)$ operator routes gradients exclusively to the underperforming branch per sample. This avoids destructive updates and roughly halves backpropagation computation for those tokens.
  • Elimination of "gradient dead zones": Visualizing loss gradients across reward space shows DPO has vast regions where gradients vanish when rewards diverge. BPO eliminates these flat regions, accelerating convergence.
  • Disproportionate gains on hard tasks: On easy problems (Level 1–2 in MATH500), DPO and BPO perform similarly. But on Level 5 problems, DPO degrades below the base model (collapsing from DCR), while BPO beats DPO by +26.1%, proving that preserving chosen likelihoods is essential for deep multi-step deduction.

Key Results

  • Overall Accuracy:
  • Llama-3.1-8B-Instruct: Jumps from 18.8% (DPO) to 28.9% (BPO), a +10.1% absolute gain.
  • Qwen2.5-Math-7B-Base: Improves from 41.0% (DPO) to 46.7% (BPO), outperforming GPT-4o (43.3%) and Qwen2.5-Math-7B-Instruct (43.8%).
  • AIME 2024: BPO achieves 30.0% on Qwen2.5-Math-7B vs. just 6.7% for standard DPO and 20.0% for Cal-DPO.
  • Outperforming DPO Variants: On Qwen2.5-Math-7B, BPO (46.7%) outperforms IPO (43.1%), SLiC (41.7%), Cal-DPO (43.6%), and DPOP (44.9%).
  • Broad Compatibility: Pairing BPO's min-margin formulation with SLiC hinge loss boosts performance from 41.7% to 46.7% (+5.0%).

Limitations and Future Work

  • Offline Only: The current formulation is strictly evaluated on offline preference pairs. The authors note that extending the balanced reward margin to on-policy exploration (where the policy interacts dynamically with a reward signal or verifier during training) remains unexplored.
  • Focus on Reasoning: Experiments are centered heavily on mathematical reasoning; further validation on open-ended chat, instruction-following, and safety alignment benchmarks would confirm whether DCR mitigation translates equally to softer conversational objectives.

Summary updated

Abstract

Direct Preference Optimization (DPO) have emerged as a popular method for aligning LLMs with human preferences. While DPO effectively preserves the relative ordering between chosen and rejected responses through pairwise ranking losses, it often neglects absolute reward magnitudes. This oversight can decrease the likelihood of chosen responses and increase the risk of generating out-of-distribution responses, leading to poor performance. We term this issue Degraded Chosen Responses (DCR).To address this issue, we propose Balanced Preference Optimization (BPO), a novel framework that dynamically balances the optimization of chosen and rejected responses through two key components: balanced reward margin and gap adaptor. Unlike previous methods, BPO can fundamentally resolve DPO's DCR issue, without introducing additional constraints to the loss function. Experimental results on multiple mathematical reasoning tasks show that BPO significantly outperforms DPO, improving accuracy by +10.1% with Llama-3.1-8B-Instruct and +11.7% with Qwen2.5-Math-7B. It also surpasses DPO variants by +3.6% over IPO , +5.0% over SLiC, and +3.1% over Cal-DPO on the same model. Remarkably, our algorithm requires only a single line of code modification, making it simple to implement and fully compatible with existing DPO-based frameworks.
BibTeX
@misc{sun2025bpo,
  author = {Lin Sun and Chuang Liu and Peng Liu and Bingyang Li and Weijia Lu and Ning Wu},
  title = {{BPO: Revisiting Preference Modeling in Direct Preference Optimization}},
  year = {2025},
  eprint = {2506.03557},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.03557},
}

Loading related content...

Figure