In Bayesian machine learning, we often face a frustrating trade-off when choosing an inference framework. On one hand, Markov Chain Monte Carlo (MCMC) methods are asymptotically exact but notoriously slow and difficult to scale. On the other hand, Variational Inference (VI) frames posterior estimation as a fast, deterministic optimization problem.
However, standard mean-field VI comes with a major catch: it forces us to assume that all latent variables and parameters are completely independent in the approximate posterior. When parameters are tightly coupled in reality, this independence assumption introduces severe approximation bias, traps optimization in poor local optima, and causes the model to underestimate uncertainty.
Collapsed Variational Inference (CVI)—often called Collapsed Variational Bayes (CVB)—resolves this tension. The core insight is elegant: analytically integrate out global parameters before applying variational independence assumptions. By optimizing the model in a "collapsed" space, we can maintain the speed of deterministic optimization while capturing the complex dependencies that standard variational methods ignore.
The Flaw in the Mean-Field Assumption
To understand why collapsing works, consider a standard latent variable model like Latent Dirichlet Allocation (LDA). The model contains global parameters $\theta$ (such as topic-word distributions) and local latent variables $z$ (such as token-to-topic assignments).
A standard mean-field variational approach approximates the true posterior $p(\theta, z | x)$ using a fully factorized distribution:
$$q(\theta, z) = q(\theta) q(z)$$
This factorization forces a strict independence between the global parameters $\theta$ and local variables $z$. If knowing a specific latent assignment $z_i$ should drastically shift our certainty about $\theta$, the mean-field approximation cannot capture this. The optimization landscape becomes riddled with local minima because the algorithm must coordinate updates between two highly correlated, explicitly parameterized spaces.
Standard Mean-Field VI:
[ Global Parameters θ ] <--- (Forced Independence) ---> [ Local Latent Variables z ]
Collapsed VI:
[ Global Parameters θ ] --- (Analytically Integrated Out)
│
▼
[ Collapsed Space of z ]
The Mechanics of Collapsing
Instead of forcing $\theta$ and $z$ to be independent, CVI leverages conjugate priors to analytically marginalize (integrate out) the global parameters $\theta$ entirely from the joint distribution:
$$p(z, x) = \int p(\theta, z, x) d\theta$$
As Teh et al. (2006) demonstrated in their foundational work on LDA, this algebraic step fundamentally changes the variational task. Because $\theta$ no longer explicitly exists in the optimization objective, we only need to define a variational distribution over the local latent variables:
$$q(z) = \prod_i q(z_i)$$
By eliminating the global parameters from the optimization variables, the model operates in a significantly lower-dimensional space. As a result, the evidence lower bound (ELBO) becomes inherently tighter. As Foulds et al. (2013) noted, uncollapsed stochastic methods often suffer from slower convergence because they must track unique variational parameters for every global and local interaction, whereas collapsing yields a tighter variational bound and superior convergence rates.
The Paradox: Why Weakening Dependencies Works
Intuitively, integrating out a shared global parameter $\theta$ should make the remaining latent variables $z$ more dependent on one another, not less. If $z_1$ and $z_2$ both drew their values from $\theta$, removing $\theta$ means $z_1$ now directly carries information about $z_2$.
Why, then, does a factorized approximation $q(z) = \prod_i q(z_i)$ work better in this collapsed space?
The answer lies in how variables interact after marginalization. As explored by Wang et al. (2013) in the context of Hidden Markov Models, integrating out global parameters induces a global dependency structure that paradoxically weakens the local pairwise dependencies between individual hidden variables.
In the collapsed space, the latent variables no longer react to a single, volatile estimate of a global parameter. Instead, as Koeppl et al. (2018) explained, they interact through a global field of aggregate sufficient statistics (such as count matrices or sum totals). In a large dataset, the contribution of any single latent variable $z_i$ to these aggregate statistics is minuscule. Because no single variable can drastically alter the global field, the conditional distribution of $z_i$ given all other variables becomes highly stable. This stability makes the independence assumption of the mean-field approximation highly accurate in the collapsed space, whereas it fails completely in the uncollapsed space.
Broad Applications Across Model Domains
The strategy of parameter marginalization has proven successful across a wide variety of probabilistic architectures, moving far beyond its origins in discrete topic modeling.
Discrete and Nonparametric Structures
In Dirichlet Process Mixture Models and Hierarchical Dirichlet Processes, marginalizing out cluster weights allows the variational framework to capture latent dependencies that standard approximations lose (Kurihara et al., 2007; Teh et al., 2007). Similarly, in complex structural domains like Probabilistic Context-Free Grammars, Wang et al. (2013) showed that integrating out rule probabilities prevents deterministic optimization from getting trapped in the poor local optima that plague standard Maximum Likelihood or uncollapsed variational methods.
Continuous and Deep Probabilistic Models
The same intuition applies to continuous spaces. In Sparse Gaussian Processes (GPs) and Deep Gaussian Processes, tracking correlations between latent inducing variables and network weights is computationally grueling.
- Deep GPs: Lindinger et al. (2020) demonstrated that analytically marginalizing correlated global latent variables reduces the variance of the ELBO gradients, facilitating more stable and faster convergence.
- State-Space Models: Fan et al. (2023) utilized an analytical collapsing technique to marginalize out inducing variables in Gaussian Process State-Space Models, allowing inference to proceed in a lower-dimensional space and avoiding the need to sample state trajectories and transition variables simultaneously.
- Hyperparameter Integration: Rather than optimizing kernel hyperparameters—which often leads to overconfident predictions—Lalchand et al. (2022) introduced a "doubly collapsed" framework that analytically integrates out latent inducing variables, simplifying the landscape so that hyperparameters can be fully integrated via MCMC without massive computational overhead.
Summary
Collapsed Variational Inference works because it changes the geometry of the optimization problem. By analytically integrating out global parameters, it transforms a highly coupled, high-dimensional optimization landscape into a smooth, lower-dimensional space where latent variables interact only through stable, aggregate statistics. By understanding this mechanism, practitioners can design variational bounds that are tighter, less sensitive to initializations, and capable of preserving the true underlying uncertainties of complex probabilistic models.
Key Foundational Work
A Collapsed Variational Bayesian Inference Algorithm for Latent Dirichlet Allocation (2006) — contributed 2 concept elements to this direction.
Collapsed Variational Inference for HDP (2007) — contributed 2 concept elements to this direction.
Collapsed Variational Dirichlet Process Mixture Models (2007) — contributed 2 concept elements to this direction.