Tuning Mode Selection via Parameterized Divergence Minimization

163 concepts101 papers

When training probabilistic models, we constantly face a fundamental geometric conflict: do we want our approximation to cover the entire mass of a target distribution, or do we want it to lock onto the most prominent modes?

Standard objectives like the Kullback-Leibler (KL) divergence force a rigid choice. Forward KL, $\text{KL}(p \parallel q)$, is mass-covering (or zero-avoiding). It penalizes the model $q$ heavily if $q(x)$ is small where the target $p(x)$ is large, forcing $q$ to stretch across all modes, which often results in a blurry, over-extended distribution. Conversely, reverse KL, $\text{KL}(q \parallel p)$, is mode-seeking (or zero-forcing). It penalizes the model if it puts mass where $p(x)$ is zero, forcing $q$ to collapse onto a single sharp mode while ignoring others.

To escape this binary choice, a powerful research direction focuses on parameterized divergence measures—such as $\alpha$-divergences, Rényi divergences, Tsallis divergences, and skewed $f$-divergences. By introducing a continuous hyperparameter, these generalized objectives allow practitioners to smoothly tune a model's behavior anywhere along the spectrum between mass-covering and mode-seeking extremes.

The Geometry of Parameterized Divergences

The core mechanism behind parameterized divergences is the ability to alter how discrepancies in different regions of the probability space are weighted. The tuning parameter (often denoted as $\alpha$) acts as a dial for density sensitivity.

As shown by Li et al., using Rényi $\alpha$-divergences yields a generalized variational bound that interpolates between standard variational inference ($\alpha \to 1$) and exact log marginal likelihood estimation ($\alpha \to 0$). Positive values of $\alpha$ encourage zero-forcing behavior, while negative values promote mass-covering behavior. This allows practitioners to balance the risk of underestimating variance against the risk of overestimating the distribution's spread.

   Mass-Covering (Zero-Avoiding)          Mode-Seeking (Zero-Forcing)
   <---------------------------------------------------------------->
   Negative α                                              Positive α
   Captures broad variance                        Concentrates on peaks
   Low precision, high recall                     High precision, low recall

This continuous interpolation unifies previously disparate inference algorithms. Neng Wan established a generalized framework demonstrating that various variational inference techniques are mathematically related members of the same unified design space. For instance, Hernández-Lobato et al. demonstrated that minimizing $\alpha$-divergences via a black-box stochastic framework allows a single objective to seamlessly transition between Variational Bayes (VB) and Expectation Propagation (EP). Similarly, Sato et al. revealed that successful hybrid inference algorithms can be explicitly constructed by applying different $\alpha$-divergence projections to distinct components of a graphical model.

Balancing Quality and Diversity in Generative Models

In generative modeling, the trade-off between mode-seeking and mass-covering maps directly to the balance between precision (individual sample quality) and recall (distribution diversity).

Verine et al. formalized this by introducing a family of $f$-divergences that explicitly optimizes specific points along the precision-recall curve during training. In text-to-image generation, Sun et al. leveraged $f$-divergence minimization to prevent models from becoming overly repetitive while maintaining alignment with human preferences.

This flexibility is particularly crucial when aligning Large Language Models (LLMs) with human feedback. Standard Direct Preference Optimization (DPO) relies on a fixed reverse-KL penalty, which can restrict the model's output variety and cause it to output repetitive answers. To solve this, Han et al. proposed $f$-PO and $\alpha$-PO, demonstrating that tuning the $\alpha$ parameter allows practitioners to control whether an LLM prioritizes the single most probable correct answer or covers a broader, more creative range of human-like responses.

For continuous generative architectures like diffusion and energy-based models, parameterized divergences provide a way to handle non-overlapping distributions and stabilize training. Jayashankar et al. introduced Score-of-Mixture Training using alpha-skew Jensen-Shannon divergences. This objective remains well-defined even when target and generated distributions lack overlap, allowing models to smoothly interpolate between sharp sample generation and broad mode coverage. In a similar vein, Shen et al. utilized Rényi diffusion models to decouple exact density estimation from perceptual sample quality.

Controlling Exploration in Decision-Making

In reinforcement learning (RL), planning, and control, the choice of divergence dictates how an agent balances exploration (mass-covering) and exploitation (mode-seeking).

Belousov et al. generalized entropic regularization in Markov Decision Processes using $f$-divergences, showing that adjusting the divergence parameter directly controls the policy's sensitivity to advantage estimates. In cognitive science and variational inference, Faccio et al. showed that swapping the standard KL divergence for a Rényi divergence provides a mathematical mechanism for artificial agents to toggle between greedy exploitation and exploratory behavior.

This principle extends directly to robotics and tree search:

  • Model Predictive Control: So et al. incorporated Tsallis divergences into Model Predictive Control (MPC), allowing controllers to flexibly weight trajectory samples by modifying the underlying entropy shape.
  • Tree Search: For planning under uncertainty, Dam et al. integrated $\alpha$-divergences into Monte-Carlo Tree Search (MCTS), using generalized power means to tune the search profile between conservative averaging and optimistic maximization.

Robustness in State Estimation

Beyond generation and control, parameterized divergences offer significant advantages in tracking and state estimation, where standard Gaussian assumptions often fail due to non-linearities or multi-modal noise.

Gultekin et al. and the authors developed non-linear Kalman filters that directly optimize $\alpha$-divergences using reparameterization gradients. By tuning $\alpha$, these filters can better capture specific peaks or maintain probability mass across disconnected regions of the parameter space, outperforming traditional Extended or Unscented Kalman Filters.

This capability is mirrored in complex physical applications; for example, Tiede et al. utilized $\alpha$-divergences and normalizing flows to map highly multi-modal astronomical posteriors without suffering from mode collapse, demonstrating that treating the divergence measure as a tunable hyperparameter fundamentally improves a model's capacity to handle real-world uncertainty.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_generalized_divergence_measures_a8bbc4cd,
  title = {generalized divergence measures},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generalized-divergence-measures/rdr_a8bbc4cd47d24a7db30f5d2dd958d121},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_generalized_divergence_measures_a8bbc4cd_v1,
  title = {generalized divergence measures},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/generalized-divergence-measures/rdr_a8bbc4cd47d24a7db30f5d2dd958d121/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake