Analytical Marginalization for Tighter Variational Bounds

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 237 papers52 conceptsfrozen 2026-10-04

In Bayesian inference, we often face a difficult choice: we want the accuracy of exact posterior distributions, but we are forced to settle for the speed of approximations. The most common approximation, Mean-Field Variational Inference (MFVI), achieves speed by assuming that all latent variables are independent. While this makes the math easy, it ignores the very correlations that define the structure of our data.

A powerful alternative has emerged across diverse domains—from topic modeling to Gaussian Processes—known as analytical marginalization (or "collapsing"). By integrating out certain variables mathematically before applying variational approximations, researchers have found they can achieve tighter bounds, more stable optimization, and better-calibrated uncertainty.

The Cost of Independence

Standard variational inference approximates a complex posterior $p(z, \theta | x)$ with a simpler distribution $q(z, \theta)$. To make this tractable, the mean-field assumption typically splits this into $q(z)q(\theta)$.

As Yee Whye Teh, Kenichi Kurihara, and Max Welling pointed out in their work on Collapsed Variational Inference for HDP, this independence assumption is a major source of bias. In reality, the global parameters $\theta$ (like topic distributions) and local variables $z$ (like word assignments) are deeply coupled. Forcing them to be independent in the approximation weakens the Evidence Lower Bound (ELBO), leading to sub-optimal models.

Collapsing the Parameter Space

The "collapsed" approach changes the order of operations. Instead of approximating the joint distribution, we use the chain rule: $p(z, \theta | x) = p(\theta | z, x) p(z | x)$. If we choose conjugate priors, we can often integrate out $\theta$ analytically.

This leaves us with a marginal distribution $p(z | x)$ that depends only on the local variables. When we apply variational inference to this "collapsed" model, we are optimizing a distribution over a much smaller space. Kenichi Kurihara and colleagues demonstrated in Collapsed Variational Dirichlet Process Mixture Models that this marginalization accounts for the dependencies that mean-field methods ignore, resulting in a tighter variational bound and faster convergence.

Why Marginalization Improves Optimization

Beyond just accuracy, marginalization fundamentally changes the "shape" of the problem we are trying to solve. There are three primary reasons why this works:

  1. Reduced Variance: In stochastic optimization, we often rely on samples to estimate gradients. Rajan Walia and Chung-chieh Shan showed that symbolically integrating out latent variables reduces the variance of these samples. This is a form of Rao-Blackwellization: by replacing a random variable with its analytical expectation, we get a cleaner signal for training.
  2. Landscape Smoothing: Perhaps the most striking intuition comes from David Wipf. In Marginalization is not Marginal, Wipf proves that integrating over the latent space acts as a selective smoothing mechanism. It "fills in" bad local minima that would otherwise trap deterministic autoencoders, while preserving the global minima that represent the most parsimonious (sparse) explanations of the data.
  3. Lower Dimensionality: By eliminating global parameters, the optimizer moves through a lower-dimensional manifold. Xuhui Fan and Terry O'Kane applied this to Gaussian Process State-Space Models, showing that marginalizing inducing variables makes inference significantly more stable and computationally efficient.

Scaling to Complex Architectures

While early work focused on discrete models like Latent Dirichlet Allocation (LDA), the principle has expanded to continuous and deep models. James R. Foulds and Max Welling developed stochastic versions of collapsed inference for big data, proving that the benefits of collapsing persist even in online learning environments.

In the world of Gaussian Processes, Michalis Titsias has shown that analytically eliminating local variational parameters can reduce noise-overestimation bias. Similarly, Jakob Lindinger and Barbara Rakitsch demonstrated that moving beyond mean-field assumptions in Deep Gaussian Processes through marginalization leads to much better-calibrated predictive uncertainties.

Even in modern generative models like VAEs, collapsing remains a vital tool. Kaspar Märtens and Christopher Yau used a collapsed variational strategy to marginalize out mixture weights, allowing their model to automatically prune unnecessary clusters and avoid the local optima that plague standard VAE training.

Summary of Intuition

Analytical marginalization works because it respects the internal structure of the model. By doing the "hard math" of integration upfront, we provide the optimizer with a smoother, lower-dimensional, and more accurate objective function. Whether it is called "collapsing," "symbolic integration," or "structured inference," the core lesson is the same: whenever you can integrate a variable out, you should. The resulting "collapsed" landscape is almost always easier to navigate than the original one.

Papers in this version

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 20 papers
  2. Version 1 · 2026-10-04 · 37 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_collapsed_variational_inference_308a9a2b,
  title = {collapsed variational inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/collapsed-variational-inference/rdr_308a9a2b2c804cb785151ac517e708ad},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_analytical_marginalization_for_tighter_v_308a9a2b_v1,
  title = {Analytical Marginalization for Tighter Variational Bounds},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/analytical-marginalization-for-tighter-variational-bounds/rdr_308a9a2b2c804cb785151ac517e708ad/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}