Introduction
In large-scale Bayesian inference, practitioners are often forced to choose between two competing methodologies for approximating complex, high-dimensional posterior distributions: Markov Chain Monte Carlo (MCMC) and Variational Inference (VI). While MCMC (such as Langevin diffusion) boasts strong theoretical backing and non-asymptotic convergence guarantees, it can be computationally sluggish and difficult to tune. Conversely, VI is generally faster and scales better by framing sampling as an optimization problem—fitting a simpler distribution to the target. However, VI severely lacks the rigorous theoretical guarantees enjoyed by MCMC.
This paper bridges the gap between the practical speed of VI and the rigorous theoretical guarantees of MCMC. By framing Gaussian VI and Gaussian mixture VI as gradient flows on the Bures-Wasserstein space, the authors provide non-asymptotic convergence guarantees and introduce a novel "Gaussian particle" method for approximating complex distributions.
The Challenge of Non-Convexity
The core issue hindering theoretical guarantees for VI is that its optimization objective—minimizing the Kullback-Leibler (KL) divergence—is typically non-convex with respect to the parameters of the approximating distribution (mean and covariance). This paper asks a fundamental question: can we equip VI with MCMC-style, non-asymptotic convergence guarantees by viewing the optimization through the right geometric lens?
Methodology: A Geometric Lens
The authors tackle this problem using the mathematics of optimal transport, specifically Wasserstein gradient flows.
- Gaussian VI via Bures-Wasserstein Geometry: The authors demonstrate that a well-known heuristic from Variational Kalman Filtering (Särkkä’s ODEs) is actually the exact gradient flow of the KL divergence constrained to the Bures-Wasserstein manifold—the space of Gaussian distributions equipped with the 2-Wasserstein distance.
- Bures-Wasserstein SGD: To make this theoretically sound and practical, they discretize this gradient flow into a novel Stochastic Gradient Descent (SGD) algorithm that operates directly on the Bures-Wasserstein space.
- Mixtures of Gaussians & Gaussian Particles: Because a single Gaussian is often too simple to capture multi-modal distributions, the framework is extended to mixtures of Gaussians. By defining a Wasserstein space over the Bures-Wasserstein space, they introduce a system of interacting "Gaussian particles." Unlike traditional particle methods where particles are mere point masses, each particle here is an entire Gaussian distribution with its own evolving mean and covariance.
Key Insights
- Hidden Convexity: The "aha moment" of the paper is the realization that even though the VI objective is non-convex in standard Euclidean space, it becomes $\alpha$-convex when viewed through the lens of Bures-Wasserstein geometry (provided the target distribution is strongly log-concave). This geometric shift unlocks the MCMC-style proofs for VI.
- Särkkä’s Heuristic is Principled: The paper provides rigorous mathematical justification for Särkkä’s ODEs, proving they are the exact Wasserstein gradient flow of the KL divergence over the space of Gaussians, rather than just a convenient engineering trick.
- Continuous Particle Systems: Traditional interacting particle systems (like Stein Variational Gradient Descent) use point masses, requiring many particles to approximate a continuous density. By using Gaussian particles, the proposed mixture model maintains a continuous probability distribution at all times, making it vastly more expressive for a given number of components.
Key Results and Validation
The paper focuses heavily on theoretical development, proving a non-asymptotic convergence rate for their Bures-Wasserstein SGD algorithm. If the target distribution is strongly log-concave, the algorithm converges to the optimal Gaussian approximation in polynomial time. Specifically, reaching an error of $\epsilon^2$ requires $\mathcal{O}\left(\frac{d}{\alpha^3 \epsilon^2} \log(1/\epsilon)\right)$ iterations, where $d$ is the dimension and $\alpha$ is the strong log-concavity parameter.
Empirically, the authors validate their algorithms using synthetic and standard toy distributions, including a 2D synthetic bimodal target and a posterior from a logistic regression problem. The ODE integration and Gaussian particle methods successfully and rapidly approximate these targets. Notably, the proposed VI methods achieve a better KL divergence approximation than standard Laplace approximations.
Limitations and Future Work
While theoretically profound, the approach has practical limitations:
- Hessian Oracle Requirement: The Bures-Wasserstein SGD algorithm requires access to the Hessian (second derivatives) of the target distribution, resulting in a higher per-iteration computational cost compared to typical MCMC samplers. Developing Hessian-free algorithms is a key area for future work.
- Strong Assumptions: The theoretical guarantees for Gaussian VI rely on the target distribution being strongly log-concave, an assumption that may not hold in complex deep learning scenarios.
- Fixed Mixture Weights: In the proposed Gaussian mixture algorithm, the weights of the mixture components remain fixed. Dynamically updating these weights—potentially using Wasserstein-Fisher-Rao geometry as sketched in the paper's appendix—remains an exciting avenue for future research.
Summary updated