Direct Preference Optimization (DPO) revolutionized language model alignment by eliminating the need for an explicit reward model. Instead of training a separate reward model and running computationally heavy reinforcement learning, DPO optimizes the language model directly using preference pairs consisting of a chosen response ($y_w$) and a rejected response ($y_l$).
However, this elegance masks a fundamental mathematical vulnerability. DPO’s objective function is under-specified: it focuses exclusively on maximizing the relative margin between the log-probabilities of the chosen and rejected responses. As Yuan et al. observe, this relative formulation lacks control over absolute probabilities.
To increase the margin $\log \pi(y_w) - \log \pi(y_l)$, the optimization algorithm has two choices: it can push the probability of the preferred response up, or it can push the probability of the rejected response down. In practice, the model frequently chooses the latter path so aggressively that the absolute log-probabilities of both responses crash toward zero (Agarwal et al.; Gupta et al.). This counter-intuitive phenomenon is known as likelihood displacement or likelihood collapse.
Why Absolute Likelihood Collapses
Why does an objective designed to favor good answers paradoxically lower their generation probability? The root causes lie in gradient dynamics and the constraints of probability space.
Gradient Entanglement
In many alignment tasks—especially reasoning, mathematics, and code generation—the preferred and rejected responses are not completely different sequences. They often share a massive overlap in tokens, differing only by a critical logical step or a final answer token. Yuan et al. describe this as gradient entanglement: because the token sequences are highly correlated, the optimization signals for both outputs are tightly coupled. When the negative gradient aggressively suppresses the rejected response, it inadvertently drags down the preferred response along with it (Li et al.).
The Squeezing Effect and Mass Leakage
A language model's output vocabulary defines a probability distribution that must sum to exactly one. When a margin-based loss forces the model to decrease the probability of a rejected sample relative to a fixed reference model, it creates what Ren terms a "squeezing effect."
If the negative pressure on the rejected response is misallocated or applied to tokens that are already unlikely, the probability mass does not automatically transfer to the preferred response. Instead, as Gorbatovski et al. and Yan et al. demonstrate, the probability mass leaks out into irrelevant, low-quality, or out-of-distribution (OOD) tokens that were never part of the training data.
Asymmetric Domination by the Rejected Response
The mathematical structure of standard preference losses makes them highly sensitive to the dispreferred example. Cho et al. point out that the optimization can become entirely dominated by minimizing the probability of rejected responses. When the model encounters a rejected response to which it originally assigned high confidence, the resulting imbalanced gradients can severely impair its generative capabilities (Xiao et al.). The model becomes excellent at knowing what not to say, but loses its ability to confidently generate the correct sequence (Fisch et al.).
The Cost of Displaced Likelihood
When absolute generation probabilities collapse, the model's performance degrades across several dimensions:
- Loss of General Reasoning: Because the model's absolute confidence in valid outputs is eroded, its performance on complex, non-targeted reasoning tasks drops sharply (Chen et al.).
- Degraded Precision: Tasks requiring exact token sequences, such as mathematical proofs or code execution, suffer heavily because the model can no longer maintain high absolute likelihoods for precise logical steps (Xiao et al.).
- Degenerate and Repetitive Outputs: As probability mass shifts away from the training distribution into high-curvature or unconstrained regions of the logit space, the model begins to output repetitive text, gibberish, or out-of-distribution tokens (Fisch et al.; Luo et al.).
Restoring Absolute Anchors
To counteract likelihood displacement, researchers are developing alignment objectives that anchor the model's absolute scale rather than relying solely on relative margins.
One prominent strategy is absolute calibration. For instance, Xiao et al. introduce an explicit anchor to prevent the absolute likelihood of preferred answers from dropping during contrastive optimization. Similarly, Saeidi et al. utilize multi-way rankings to balance preference learning with behavioral cloning terms, mitigating the conflict that drives down preferred probabilities.
Another approach modifies the geometry of the loss landscape. Luo et al. employ sharpness-aware minimization in the logit space to smooth out high-curvature regions, preventing negative gradients from scattering probability mass into irrelevant tokens. To address the rigidity of a static reference model, Gorbatovski et al. propose dynamically updating the reference policy during training, which prevents the active policy from drifting into over-optimization and experiencing a confidence crash.
By shifting the alignment paradigm from purely relative margin maximization to calibrated, absolute probability management, these methods stabilize preference learning and preserve the underlying generative capabilities of large language models.
Key Foundational Work
Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive (2024) — contributed 7 concept elements to this direction.
Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization (2024) — contributed 3 concept elements to this direction.
A Common Pitfall of Margin-based Language Model Alignment: Gradient Entanglement (2024) — contributed 3 concept elements to this direction.