Beyond Gaussian Approximations in Variational Inference

500 concepts406 papers1 proposal

Variational Inference (VI) frames the challenge of Bayesian posterior estimation as an optimization problem. Instead of simulating a posterior distribution using computationally expensive Markov Chain Monte Carlo (MCMC) sampling, we select a family of tractable distributions $q(z)$ and find the member that minimizes the Kullback-Leibler (KL) divergence to the true posterior $p(z|x)$.

For years, the default choice for this variational family has been a fully factorized (mean-field) Gaussian distribution. This choice underpins foundational algorithms like Bayes by Backprop, introduced by Charles Blundell et al. in Weight Uncertainty in Neural Network. Gaussians are mathematically convenient and perfectly compatible with the reparameterization trick. However, this simplicity introduces a severe expressivity gap. Real-world posteriors—especially in deep neural networks, non-conjugate models, and complex physical systems—are rarely symmetric, unimodal, or neatly uncoupled. They feature multi-modality, heavy tails, and intricate non-linear dependencies.

When we force a complex, multi-modal posterior landscape into a rigid Gaussian mold, the optimization process underestimates variance, ignores alternative modes, and provides overconfident, biased uncertainty estimates. As noted by Michalis K. Titsias et al. in Doubly Stochastic Variational Bayes for non-Conjugate Inference, overcoming these parametric constraints remains a central challenge for capturing non-standard posterior shapes.

To close this expressivity gap, researchers have developed several distinct strategies to construct highly flexible variational families that remain computationally tractable.

Warping Space with Normalizing Flows

How can we make a variational distribution flexible enough to capture complex geometries while keeping its density function calculable? The core intuition behind Normalizing Flows is to take a simple base distribution (such as a standard isotropic Gaussian) and pass it through a sequence of invertible, differentiable transformations.

Imagine the base distribution as a lump of clay. Each transformation in the sequence stretches, squeezes, and folds this clay across the coordinate space. Because each transformation is invertible and differentiable, we can use the multivariate change-of-variables theorem to compute the exact probability density of the final, warped shape.

In their seminal paper, Variational Inference with Normalizing Flows, Danilo J. Rezende et al. demonstrated that by stacking these transformations, one can construct arbitrarily complex, multi-modal probability densities. This allows the variational distribution to mold itself precisely to the true posterior landscape while maintaining the scalability required for high-dimensional problems.

Auxiliary Variables and Implicit Samplers

Another way to break free from simple parametric families is to expand the space or eliminate the requirement for an analytical density function altogether.

Adding Dimensions: Auxiliary Models

Instead of directly enriching the distribution of the latent variables $z$, we can introduce auxiliary latent variables $u$. By defining a joint variational distribution $q(z, u)$, the marginal distribution $q(z) = \int q(z, u) du$ can become highly complex and multi-modal, even if the joint distribution itself is relatively simple.

The authors of Auxiliary Deep Generative Models, Lars Maaløe et al., showed that this expansion significantly boosts the expressivity of the inference model. It allows the system to capture intricate dependencies in high-dimensional data without complicating the underlying generative process.

Dropping the Density: Implicit Models

What if we abandon analytical density formulas entirely? An implicit variational properties framework defines the variational distribution not by its formula, but by a generative mechanism—a neural network that takes random noise as input and outputs a sample.

Because we cannot write down the log-density $\log q(z)$ for such a model, we cannot compute the standard KL divergence directly. To solve this, Lars M. Mescheder et al. introduced Adversarial Variational Bayes: Unifying Variational Autoencoders and Generative Adversarial Networks. They utilized an adversarial discriminator to estimate the divergence between the approximate posterior and the prior, bypassing the need for an analytical density function.

Similarly, Dustin Tran et al. developed Hierarchical Implicit Models and Likelihood-Free Variational Inference, which combines the flexibility of implicit simulator-based models with hierarchical Bayesian reasoning, allowing for highly expressive posterior approximations over massive datasets.

Particle-Based Non-Parametric Inference

Instead of defining a continuous distribution function via neural networks, we can represent the variational distribution non-parametrically as a discrete cloud of evolving particles.

This is the foundational concept behind Stein Variational Gradient Descent (SVGD), proposed by Qiang Liu 0001 et al. in Stein Variational Gradient Descent: A General Purpose Bayesian Inference Algorithm. SVGD does not assume a specific parametric shape like a Gaussian. Instead, it iteratively updates a set of particles to match the target distribution by driving them along a trajectory determined by a functional gradient.

To prevent all particles from collapsing into the nearest local mode, SVGD uses a repulsive force derived from a kernel function. This force pushes the particles apart, ensuring they spread out to map the entire geometry and capture multiple modes of the target distribution.

The Geometric Perspective

Expanding the expressivity of variational families also requires rethinking the geometry of the optimization space itself. Traditional variational inference treats parameters as vectors in a flat Euclidean space. However, when navigating complex probability distributions, optimization can be accelerated by leveraging the natural, curved geometry of probability manifolds.

For example, Michael Ziyang Diao et al. demonstrated in Forward-Backward Gaussian Variational Inference via JKO in the Bures-Wasserstein Space that optimizing distributions within the Bures-Wasserstein space provides strong theoretical guarantees for finding stationary points, even in non-log-concave settings.

Even when using local approximations like the Laplace method, accounting for global curvature is vital. Standard Laplace approximations fit a symmetric Gaussian at a single mode, failing when the true posterior is skewed or constrained.

To resolve this, Hanlin Yu et al. developed the Riemannian Laplace Approximation with the Fisher Metric. By incorporating the Fisher Information Matrix as a Riemannian metric, the authors corrected the systematic underestimation of uncertainty caused by rigid Euclidean assumptions, allowing local approximations to respect the underlying geometry of the parameter space.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 425 papers
  2. Version 1 · 2026-10-04 · 294 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_variational_family_expressivity_df57c25a,
  title = {variational family expressivity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/variational-family-expressivity/rdr_df57c25a49f640828095bb2b7971f253},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_variational_family_expressivity_df57c25a_v2,
  title = {variational family expressivity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/variational-family-expressivity/rdr_df57c25a49f640828095bb2b7971f253/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake