In Bayesian machine learning, we often face the challenge of approximating a complex, "true" posterior distribution $p$ with a simpler, tractable distribution $q$. The standard tool for this is Variational Inference (VI), which typically minimizes the reverse Kullback-Leibler (KL) divergence, $KL(q \| p)$.
While computationally convenient, reverse KL has a notorious "personality flaw": it is mode-seeking. If the true distribution has multiple peaks, a reverse-KL-optimized model will often collapse onto just one of them, completely ignoring the others. In fields like gravitational-wave physics or autonomous vehicle safety, this "mode collapse" isn't just a mathematical quirk—it’s a failure to account for all possible physical realities or failure modes.
To solve this, a growing body of research advocates for mass-covering (or inclusive) objectives, such as the forward KL divergence, $KL(p \| q)$, and broader $\alpha$-divergences. These objectives prioritize covering the entire support of the target distribution, ensuring that no region of high probability is left behind.
The Intuition of Zero-Avoiding
The difference between mode-seeking and mass-covering behavior stems from how these divergences handle regions where one distribution is zero and the other is not.
In the standard reverse KL, $KL(q \| p) = \int q(x) \log \frac{q(x)}{p(x)} dx$, the objective is "zero-forcing." If $p(x)$ is near zero, $q(x)$ must also be near zero to keep the ratio from exploding. This encourages $q$ to "shrink-wrap" around a single mode of $p$ to avoid the high penalty of placing mass where $p$ has none.
The forward KL, $KL(p \| q) = \int p(x) \log \frac{p(x)}{q(x)} dx$, flips this logic. It is zero-avoiding. If $p(x)$ is large, $q(x)$ must be large to avoid a massive penalty. As Daniel C. Hackett and colleagues demonstrate in Flow-based sampling for multimodal and extended-mode distributions in lattice field theory, this property forces the model to spread its "mass" across all modes of the target, providing a much more robust representation of the true uncertainty.
Scaling to Complex Physics
The shift toward mass-covering objectives has been particularly transformative in the physical sciences. When analyzing gravitational waves, missing a potential parameter solution could lead to incorrect astrophysical conclusions. Maximilian Dax and Stephen R. Green show in Neural Importance Sampling for Rapid and Reliable Gravitational-Wave Inference that training neural proposals with forward KL ensures the resulting distributions are mass-covering. This prevents the model from ignoring physically valid regions of the parameter space.
Similarly, in astronomy, Jeffrey Regier has applied these concepts to "deblend" overlapping stars in crowded images. In Variational Inference for Deblending Crowded Starfields, the authors argue that minimizing forward KL prevents the model from getting stuck in local minima where light sources are simply missed—a common failure of standard mode-seeking VI.
Beyond KL: The $\alpha$-Divergence Spectrum
The forward and reverse KL are actually two points on a broader spectrum of $\alpha$-divergences.
- When $\alpha \to 0$, we get the mode-seeking reverse KL.
- When $\alpha = 1$, we get the mass-covering forward KL.
- When $\alpha = 2$, we get the $\chi^2$ divergence, which is even more aggressive about covering the tails of the distribution.
Adji Bousso Dieng and David Blei introduced CHIVI in Variational Inference via $\chi$ Upper Bound Minimization, which uses the $\chi^2$ divergence to "sandwich" the true model evidence. Because $\chi^2$ is zero-avoiding, the resulting approximations are typically over-dispersed rather than under-dispersed, correcting the systematic underestimation of variance found in standard VI.
This "over-dispersed" property is a feature, not a bug. In Dropout Inference in Bayesian Neural Networks with Alpha-divergences, Yingzhen Li and Yarin Gal show that using mass-covering $\alpha$-divergences prevents models from becoming overconfident, which is critical for detecting out-of-distribution data or adversarial attacks.
The Synergy with Importance Sampling
One of the most powerful applications of mass-covering models is as proposal distributions for Importance Sampling (IS).
Importance sampling works best when the proposal distribution $q$ has "heavier tails" than the target $p$. If $q$ misses a mode of $p$, the importance weights can become dangerously high-variance or even infinite. By training a model with a mass-covering objective like forward KL or forward $\chi^2$, researchers create an ideal proposal.
Chengrui Li and Anqi Wu demonstrate this in Forward $\chi^2$ Divergence Based Variational Importance Sampling, where they use a numerically stable log-space optimization to ensure the proposal covers the entire support of the target. This hybrid approach combines the speed of neural networks with the statistical guarantees of importance sampling.
Challenges in High Dimensions
While mass-covering objectives offer better reliability, they are not a free lunch. Tomas Geffner and Justin Domke point out a "curse of dimensionality" in On the difficulty of unbiased alpha divergence minimization. They show that for $\alpha > 0$, the signal-to-noise ratio of gradient estimators can decay exponentially with the number of dimensions.
This suggests that while mass-covering objectives are conceptually superior for capturing diversity, the field continues to innovate on amortized and flow-based architectures—such as the Flow Annealed Importance Sampling Bootstrap (FAB) by Laurence Illing Midgley and Vincent Stimper—to make these inclusive objectives practical for the high-dimensional problems of modern science.