In probabilistic machine learning, we rarely optimize our true objectives directly. Whether we are training a Variational Autoencoder, fitting a Sparse Gaussian Process, or estimating the divergence between two data distributions, the exact quantities we care about are usually mathematically intractable. Instead, we rely on variational frameworks to optimize a proxy—typically a lower bound on the true marginal log-likelihood (such as the Evidence Lower Bound, or ELBO) or a variational representation of an $f$-divergence.
However, these proxies introduce an approximation gap. If a variational bound is loose, maximizing it does not guarantee that we are maximizing the true likelihood or accurately aligning distributions. This loose fit leads to severe practical failure modes: models underfit, hyperparameters are incorrectly tuned, and estimators systematically underestimate distributional discrepancies.
A major research direction focuses on systematically closing this gap. By restricting mathematical dual spaces to valid probability measures, relaxing rigid structural assumptions, and minimizing estimator variance, researchers have developed tighter variational representations that fundamentally improve model expressivity and optimization reliability.
Restricting the Dual Space to Probability Measures
To understand why variational bounds can be systematically loose, we must look at how we represent distances between distributions. Many variational estimators of $f$-divergences rely on Fenchel duality, which transforms a difficult divergence estimation into an optimization problem over a space of functions.
The catch is that standard formulations allow this optimization to span a massive, unconstrained space of all integrable functions. This space includes functions that do not correspond to valid probability distributions—meaning they do not integrate to one. Because the mathematical supremum is taken over an overly broad space, the resulting lower bounds are systematically loose, leading to a severe underestimation of the true divergence when calculated from finite data samples.
The intuition for fixing this is straightforward: if we know our target targets are probability distributions, we should force our variational optimization space to reflect that constraint.
Standard Variational Space: [ All Integrable Functions (Unconstrained) ] --> Loose Bound
▲
Restrict
│
Refined Variational Space: [ Valid Probability Measures (Sum to 1) ] --> Tighter Bound
As demonstrated by Ruderman et al. in Tighter Variational Representations of f-Divergences via Restriction to Probability Measures, explicitly incorporating the constraint that probability measures must integrate to one into the variational representation yields a strictly tighter lower bound. This simple mathematical restriction eliminates the systematic underestimation inherent in traditional estimators.
This insight has powerful downstream effects across learning theory and generative modeling:
- Generalization Bounds: Ohnishi et al. showed that restricting the optimization space to valid probability measures yields tighter change of measure inequalities. This directly sharpens the complexity terms used in PAC-Bayesian generalization bounds for bounded-variance distributions.
- Distributional Metrics: Agrawal et al. applied this restriction to refine the mathematical bounds between $f$-divergences and Integral Probability Metrics (IPMs), proving that constraining the space to probability measures yields a strictly sharper representation than classic alternatives like the standard Donsker-Varadhan formulation.
Relaxing the Conditional Prior Assumption
A second major source of looseness stems from structural constraints placed on the variational distributions themselves. This is most visible in Sparse Variational Gaussian Processes (SVGPs), which use a small set of $M$ "inducing variables" to approximate a dataset of $N$ points.
To keep the math convenient, standard SVGPs make a rigid assumption: they force the approximate conditional posterior distribution of the training data (given the inducing variables) to exactly match the prior conditional distribution.
This assumption is highly restrictive. It assumes the data behaves exactly like the prior, preventing the model from learning localized variations. This forces the standard ELBO to include a harsh trace-based regularization term that penalizes deviations from the prior. As a result, the model suffers from severe underfitting and a distinct failure mode: it overestimates the observation noise variance because it mistakes its own approximation errors for true data noise.
To resolve this, Titsias et al. in New Bounds for Sparse Variational Gaussian Processes proposed relaxing this decade-old constraint. By introducing additional variational parameters that allow the approximate posterior to deviate from the conditional prior, they derived a provably tighter ELBO. This new objective replaces the problematic trace regularization with a log-sum term, forcing the model to accurately distinguish between approximation error and actual observation noise without increasing the asymptotic computational complexity.
Concurrently, Bui et al. in Tighter sparse variational Gaussian processes demonstrated that allowing the approximate posterior to have smaller variances at training locations provides the exact flexibility needed to fit complex data accurately.
A closely related strategy for tightening structural bounds is "collapsing" out redundant variables entirely:
- Sum-Product Networks: Zhao et al. showed that analytically collapsing local variables results in a deterministic objective function that provides a strictly tighter lower bound on log-likelihood than the standard ELBO, significantly boosting density estimation.
- Bayesian Neural Networks: Tomczak et al. showed that collapsing out prior parameters prevents deep models from prematurely pruning useful weights. Because this collapsed bound is a highly accurate proxy for true test performance, it serves as a reliable training signal for hyperparameter optimization, eliminating the need for expensive validation loops.
Minimizing Estimator Variance and Alternative Objectives
The final frontier in tightening variational representations involves the mathematical formulation of the objectives themselves and the variance of their estimators.
There is a deep, fundamental relationship between how much an estimator fluctuates and how loose its variational bound is. As proven by Maddison et al. in Filtering Variational Objectives, the approximation gap between a variational lower bound and the true log-likelihood is approximately proportional to half the relative variance of the marginal likelihood estimator:
$$\text{Approximation Gap} \approx \frac{1}{2} \mathbb{V}_{\text{rel}}[\hat{Z}]$$
By utilizing Sequential Monte Carlo (particle filtering) to dramatically reduce the variance of the estimator $\hat{Z}$, the authors directly tightened the objective function, allowing sequential latent variable models to capture complex temporal structures in speech and music.
Beyond variance reduction, researchers have looked to alternative divergence families to construct sharper objectives. Shen et al. introduced Rényi Diffusion Models, demonstrating that a variational bound derived from Rényi divergence provides a mathematically tighter estimate of the true data distribution than the standard KL-based ELBO for specific parameter ranges, leading to state-of-the-art log-likelihood scores.
To make these bounds even sharper during training, Cheng et al. introduced learnable monotone transformations directly into the variational lower bound, allowing the model to dynamically tighten its discrepancy estimates during domain adaptation. This aligns with the broader framework proposed by Tao et al., whose Generalized Evidence Lower Bound (GLBO) leverages flexible evidence functions to provide a provably sharper approximation of marginal likelihood.
Ultimately, the choice of variational representation dictates the stability of the entire system. The Donsker-Varadhan (DV) representation of KL divergence has emerged as a preferred tool for building high-precision estimators. Ahuja utilized the DV representation within a kernel machine framework to transform non-convex neural divergence estimation into a well-behaved, convex optimization task.
In generative modeling, Caucheteux et al. demonstrated that integrating the DV representation into generative Wasserstein flows offers a tighter, more stable alternative to standard $f$-divergences. By ensuring the variational bound is as tight as possible, the error in downstream tasks—such as estimating score functions in energy-based models (Bao et al.) or improving the expressivity of normalizing flows (Huang et al.)—is minimized, leading to more stable training and superior generative quality.
Key Foundational Work
Tighter Variational Representations of f-Divergences via Restriction to Probability Measures (2012) — contributed 2 concept elements to this direction.
Barrier Frank-Wolfe for Marginal Inference (2015) — contributed 1 concept elements to this direction.