Abstract
Along with Markov chain Monte Carlo (MCMC) methods, variational inference (VI) has emerged as a central computational approach to large-scale Bayesian inference. Rather than sampling from the true posterior $\pi$, VI aims at producing a simple but effective approximation $\hat \pi$ to $\pi$ for which summary statistics are easy to compute. However, unlike the well-studied MCMC methodology, algorithmic guarantees for VI are still relatively less well-understood. In this work, we propose principled methods for VI, in which $\hat \pi$ is taken to be a Gaussian or a mixture of Gaussians, which rest upon the theory of gradient flows on the Bures--Wasserstein space of Gaussian measures. Akin to MCMC, it comes with strong theoretical guarantees when $\pi$ is log-concave.
Introduction
In Bayesian machine learning, characterizing the posterior distribution—the probability of model parameters given observed data—is a fundamental challenge. Traditionally, practitioners have relied on two distinct paradigms: Markov Chain Monte Carlo (MCMC) and Variational Inference (VI). MCMC methods are prized for their asymptotic accuracy and rigorous theoretical guarantees but are often computationally prohibitive for large-scale datasets. Conversely, VI transforms inference into an optimization problem, typically fitting a simpler distribution (such as a Gaussian) to the complex posterior. While VI is significantly faster and widely used in industry, it has historically lacked the non-asymptotic convergence guarantees that make MCMC reliable. This paper bridges this gap by framing VI as a gradient flow in the Wasserstein space, providing a rigorous foundation for its performance.
The Geometry of Variational Inference
The authors leverage the geometry of Optimal Transport to analyze the optimization landscape of VI. Specifically, they view the space of Gaussian distributions as a manifold known as the Bures-Wasserstein space.
A key "Aha!" moment in this research is the realization that while VI optimization may appear non-convex in standard Euclidean coordinates (mean and covariance), it becomes "displacement convex" when viewed through the lens of Wasserstein geometry. This geometric perspective allows the authors to prove that the evolution of a Gaussian approximation is mathematically equivalent to a constrained version of Langevin diffusion, the physical process underlying many MCMC methods.
Principled Algorithms for VI
The theoretical framework leads to the development of several principled algorithms:
- Bures-Wasserstein SGD: The authors propose a Stochastic Gradient Descent (SGD) algorithm that operates directly on the Bures-Wasserstein manifold. To maintain stability during the optimization of covariance matrices, the algorithm employs a specialized "clipping" operator.
- Gaussian Particles: To address multi-modal distributions where a single Gaussian is insufficient, the theory is extended to Mixtures of Gaussians. In this approach, each component of the mixture acts as a "particle" that interacts with others, evolving according to a system of Ordinary Differential Equations (ODEs). Unlike standard particle filters that use discrete point masses, these "Gaussian particles" provide a continuous and smooth approximation at every step.
Empirical Results and Convergence
The proposed methods were validated using synthetic bimodal targets and Bayesian logistic regression benchmarks with dimensions up to $d=100$.
Key Insights
- Theoretical Milestones: The paper presents Theorem 4, which provides a non-asymptotic, polynomial-time convergence guarantee for Gaussian VI. This result puts VI on equal theoretical footing with modern MCMC analysis for log-concave distributions.
- Superior Accuracy: In high-dimensional logistic regression tasks, the Bures-Wasserstein VI method consistently achieved lower KL divergence than the Laplace Approximation, a common "quick and dirty" baseline.
- Efficiency: The ODE-based implementation for Gaussian mixtures demonstrated remarkable speed, often converging in fewer than 30 steps, making it highly suitable for practical engineering applications.
Limitations and Future Directions
While this work marks a significant advance, several challenges remain. The most robust versions of the proposed SGD algorithm require calculating the Hessian (second derivative) of the target potential, which can be computationally intensive for complex models. Furthermore, while the convergence proofs are rigorous for log-concave distributions, extending these guarantees to highly non-convex, multi-modal landscapes is an ongoing area of research. Finally, the authors suggest that further empirical investigation into dynamic mixture weights via Wasserstein-Fisher-Rao (WFR) flows will be a valuable direction for future work.
Summary updated