TL;DR
Standard Direct Preference Optimization (DPO) often increases the gap between chosen and rejected responses by driving down the probability of both, a failure mode dubbed Degraded Chosen Responses (DCR). Balanced Preference Optimization (BPO) fixes this with a single line of code by replacing the relative reward margin $(r_w - r_l)$ with a balanced margin $\min(r_w, -\alpha r_l)$. This simple change keeps chosen response likelihoods growing and delivers massive gains in complex mathematical reasoning (+10.1% on Llama-3.1-8B and +11.7% on Qwen2.5-Math-7B).
What Problem Does This Paper Address?
Direct Preference Optimization (DPO) has largely replaced complex RLHF pipelines by directly optimizing language model probabilities using preference pairs (chosen $y_w$ vs. rejected $y_l$). DPO maximizes the relative margin:
$$\rho_\theta = r_w - r_l = \beta \log \frac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}$$
However, direct preference methods suffer from an inherent asymmetry in task difficulty:
- Lowering the likelihood of an incorrect answer $y_l$ is easy (the model can allocate probability mass to almost any alternative token).
- Raising the likelihood of a correct, multi-step derivation $y_w$ is hard (the model must pinpoint precise tokens).
As a result, gradient descent takes the path of least resistance: it aggressively pushes down $r_l$ while allowing $r_w$ to drift into negative territory. The authors term this Degraded Chosen Responses (DCR). When absolute probabilities collapse for both chosen and rejected answers, the model inadvertently boosts out-of-distribution (OOD) generation, leading to hallucinations and poor reasoning performance.
Methods
Rather than adding ad-hoc penalty terms or regularizers to the loss function (like DPOP or Cal-DPO), BPO reformulates the reward margin itself:
-
Balanced Reward Margin: The standard margin is replaced with: $$\rho^b_\theta = \min(r_w, -\alpha r_l)$$ This forces the optimizer to dynamically prioritize whichever component is falling behind. If chosen reward $r_w$ is too low ($r_w < -\alpha r_l$), gradients only update the chosen response. Once chosen responses are sufficiently rewarded, optimization shifts to suppressing rejected responses.
-
Gap Adaptor ($\alpha$): A hyperparameter $\alpha \in (0, 1]$ tunes the relative weight between reinforcing correct answers and penalizing incorrect ones. Empirical tests show that moderately relaxing the penalty on rejected responses (e.g., $\alpha \approx 0.3 \text{ to } 0.5$) yields optimal performance.
- Plug-and-Play Generality: BPO requires changing literally one line of code in existing loss implementations and works across multiple loss families (Bradley-Terry logistic loss, SLiC hinge loss, IPO squared loss, etc.).
Mathematically, BPO guarantees a lower bound on the chosen response probability:
$$\pi_\theta(y_w|x) \ge \exp\left(\frac{\gamma}{\beta}\right) \pi_{\text{ref}}(y_w|x)$$
ensuring that the policy never degrades chosen completions relative to the reference model.
Datasets
- Training: Trained on Open Reasoner Zero 57k, an open-source mathematical dataset combining subsets from AIME, MATH, Numina-Math, Tulu3 MATH, and OpenR1-Math-220k. Preference pairs were constructed using a max-min outcome reward strategy over 8 sampled trajectories per prompt.
- Evaluation: Evaluated on five competition-level math benchmarks via pass@1 accuracy:
- AIME 2024
- MATH500
- AMC 2023
- MinervaMath
- OlympiadBench
Key Insights and Ideas
- Gradients update only the weaker side: Unlike DPO, which always backpropagates through both $y_w$ and $y_l$, BPO's $\min(\cdot)$ operator routes gradients exclusively to the underperforming branch per sample. This avoids destructive updates and roughly halves backpropagation computation for those tokens.
- Elimination of "gradient dead zones": Visualizing loss gradients across reward space shows DPO has vast regions where gradients vanish when rewards diverge. BPO eliminates these flat regions, accelerating convergence.
- Disproportionate gains on hard tasks: On easy problems (Level 1–2 in MATH500), DPO and BPO perform similarly. But on Level 5 problems, DPO degrades below the base model (collapsing from DCR), while BPO beats DPO by +26.1%, proving that preserving chosen likelihoods is essential for deep multi-step deduction.
Key Results
- Overall Accuracy:
- Llama-3.1-8B-Instruct: Jumps from 18.8% (DPO) to 28.9% (BPO), a +10.1% absolute gain.
- Qwen2.5-Math-7B-Base: Improves from 41.0% (DPO) to 46.7% (BPO), outperforming GPT-4o (43.3%) and Qwen2.5-Math-7B-Instruct (43.8%).
- AIME 2024: BPO achieves 30.0% on Qwen2.5-Math-7B vs. just 6.7% for standard DPO and 20.0% for Cal-DPO.
- Outperforming DPO Variants: On Qwen2.5-Math-7B, BPO (46.7%) outperforms IPO (43.1%), SLiC (41.7%), Cal-DPO (43.6%), and DPOP (44.9%).
- Broad Compatibility: Pairing BPO's min-margin formulation with SLiC hinge loss boosts performance from 41.7% to 46.7% (+5.0%).
Limitations and Future Work
- Offline Only: The current formulation is strictly evaluated on offline preference pairs. The authors note that extending the balanced reward margin to on-policy exploration (where the policy interacts dynamically with a reward signal or verifier during training) remains unexplored.
- Focus on Reasoning: Experiments are centered heavily on mathematical reasoning; further validation on open-ended chat, instruction-following, and safety alignment benchmarks would confirm whether DCR mitigation translates equally to softer conversational objectives.
Summary updated