Generalized Divergence Minimization for Preference Optimization

21 concepts14 papers1 proposal

When aligning large language models (LLMs) with human preferences, Direct Preference Optimization (DPO) has become a standard paradigm. DPO avoids the complexity of training a separate reward model by mathematically substituting the reward function directly with the policy's log-likelihood ratio. However, standard DPO relies on a rigid foundation: it enforces a reverse Kullback-Leibler (KL) divergence penalty to keep the model close to a reference policy, and it maps preference probabilities using a standard logistic loss.

Recent research reveals that this formulation is merely a single point in a vast, continuous landscape of alignment objectives. By generalizing the distance metrics used to regularize policies and the loss functions used to process human choices, we can build a unified framework for preference optimization. This generalization is not just a theoretical exercise; it provides precise control over the fundamental trade-off between satisfying human preferences and preserving generative diversity.

Breaking the Mode-Seeking Trap with $f$-Divergences

Why does standard DPO sometimes cause a model to lose its creative edge or suffer from repetitive outputs? The answer lies in the geometry of the reverse KL divergence, $\mathbb{D}_{KL}(\pi \| \pi_{ref})$. Reverse KL is notoriously mode-seeking. It places an exceptionally high penalty on the model ($\pi$) if it assigns probability mass where the reference policy ($\pi_{ref}$) has none, but it cares much less if the model completely ignores some modes of the reference distribution. This forces the model to collapse onto a few safe, high-probability responses.

To escape this trap, researchers have proposed replacing the reverse KL constraint with the broader family of $f$-divergences. An $f$-divergence generalizes distribution distance using a convex function $f$. By choosing different forms for $f$, we can smoothly transition from mode-seeking to mean-seeking behavior.

For instance, Dongyoung Go et al. introduced this concept in Aligning Language Models with Preferences through $f$-divergence Minimization, demonstrating that minimizing alternative metrics like the Jensen-Shannon (JS) divergence yields a universal gradient formula. This formula acts as a pseudo-reward based on the ratio between the current and target distributions, significantly improving generative diversity. Building on this, Chaoqi Wang et al. developed Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints, showing that developers can leverage forward KL or JS divergence to fine-tune the exact balance between preference alignment and output variety without losing the efficiency of an offline, supervised objective.

This $f$-divergence minimization paradigm has proven to be a powerful unifying lens. Jiaqi Han et al. demonstrated in $f$-PO: Generalizing Preference Optimization with $f$-divergence Minimization that preference optimization can be viewed fundamentally as a distribution-matching task. By selecting different divergence functions (such as Hellinger distance or $\alpha$-divergences), various alignment objectives emerge naturally as distinct geometric paths toward an ideal target policy.

Reshaping Loss Landscapes with Bregman Divergences

While $f$-divergences generalize the constraint on the policy space, another line of work generalizes the optimization step itself by altering the loss function geometry. Standard preference optimization uses a logistic loss derived from the Bradley-Terry preference model. However, this treats all preference pairs with a fixed mathematical sensitivity, regardless of how confident or ambiguous the human feedback might be.

By applying mirror descent principles and replacing the standard KL divergence with Bregman divergences, we can construct a searchable space of potential loss functions. As explored by Carlo Alfano et al. in Learning Loss Landscapes in Preference Optimization and Meta-Learning Objectives for Preference Optimization, Bregman divergences define a custom geometry for model updates. This framework led to the discovery of Mirror Preference Optimization (MPO) and algorithms like TeMPO, which optimize the trajectory of the policy across training steps.

Similarly, Yeongmin Kim et al. leveraged this geometric flexibility in Preference Optimization by Estimating the Ratio of the Data Distribution. By substituting the standard logistic regression loss with broader Bregman distance measures, practitioners gain fine-grained control over how the model prioritizes different preference samples. For example, a model can be tuned to be highly sensitive to clear, confident human preferences while remaining robust to noisy or ambiguous training data.

A Grand Unification of the Alignment Zoo

The true power of this generalized perspective is its ability to demystify the rapidly growing "zoo" of alignment algorithms. Seemingly disparate methods—ranging from online reinforcement learning (RLHF) to offline supervised objectives like Identity Preference Optimization (IPO) or Kahneman-Tversky Optimization (KTO)—can be mathematically unified.

The theoretical foundations of this unification operate on multiple levels:

Extending the Boundaries of Preference Learning

As the mathematical foundations of preference optimization have become more generalized, the scope of their application has expanded beyond traditional boundaries.

First, the paradigm has broken free from the restriction of comparing two responses to the same prompt. Hritik Bansal et al. introduced Joint Preference Optimization (JPO) in Comparing Bad Apples to Good Oranges: Aligning Large Language Models via Joint Preference Optimization and its subsequent work Comparing Bad Apples to Good Oranges Aligning Large Language Models via Joint Preference Optimization. JPO generalizes standard conditional preference optimization by allowing the input instructions to vary across compared pairs. This enables models to learn from cross-contextual human judgments, capturing a broader and more holistic spectrum of decision-making while maintaining backward compatibility with standard single-prompt protocols.

Second, this generalized divergence framework is expanding into multimodal domains. Haoyuan Sun et al. applied these principles to text-to-image generation in Generalizing Alignment Paradigm of Text-to-Image Generation with Preferences Through f-Divergence Minimization. By replacing standard reverse KL constraints with generalized $f$-divergences, the authors demonstrated that text-to-image models can be aligned with human aesthetic preferences while successfully preserving the rich visual diversity of the original image distribution.

Key Foundational Work

Generalizing Offline Alignment Theoretical Paradigm with Diverse Divergence Constraints (2024) — contributed 4 concept elements to this direction.
On the Essence and Prospect: An Investigation of Alignment Approaches for Big Models (2024) — contributed 2 concept elements to this direction.
Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints (2024) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_generalized_preference_optimization_34962f30,
  title = {generalized preference optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generalized-preference-optimization/rdr_34962f3079a843e9a74a0249656c1d15},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_generalized_preference_optimization_34962f30_v1,
  title = {generalized preference optimization},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generalized-preference-optimization/rdr_34962f3079a843e9a74a0249656c1d15/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake