Abstract
Mean-field variational inference (MFVI) is a widely used method for approximating high-dimensional probability distributions by product measures. This paper studies the stability properties of the mean-field approximation when the target distribution varies within the class of strongly log-concave measures. We establish dimension-free Lipschitz continuity of the MFVI optimizer with respect to the target distribution, measured in the 2-Wasserstein distance, with Lipschitz constant inversely proportional to the log-concavity parameter. Under additional regularity conditions, we further show that the MFVI optimizer depends differentiably on the target potential and characterize the derivative by a partial differential equation. Methodologically, we follow a novel approach to MFVI via linearized optimal transport: the non-convex MFVI problem is lifted to a convex optimization over transport maps with a fixed base measure, enabling the use of calculus of variations and functional analysis. We discuss several applications of our results to robust Bayesian inference and empirical Bayes, including a quantitative Bernstein–von Mises theorem for MFVI, as well as to distributed stochastic control.
Introduction
Mean-Field Variational Inference (MFVI) serves as a cornerstone of modern Bayesian computation, enabling the approximation of complex, high-dimensional posterior distributions with simpler, independent product measures. Despite its widespread adoption, the theoretical landscape of MFVI has been historically challenging to navigate. The primary obstacle is the non-convexity of the set of product measures, which often leads to multiple local optima and complicates sensitivity analysis.
A critical question for practitioners is the stability of the MFVI solution: if the underlying model, prior, or data undergo slight perturbations, does the resulting variational approximation shift predictably or erratically? This paper provides the first quantitative stability analysis for MFVI, establishing that for strongly log-concave target distributions, the variational optimizer is both Lipschitz continuous and differentiable with respect to the target potential.
Methodology: The Lifting Trick
To overcome the inherent non-convexity of the MFVI problem, the authors introduce a novel framework based on linearized optimal transport.
Traditionally, MFVI seeks to minimize the Kullback-Leibler (KL) divergence between a product measure and a target distribution. The authors instead parametrize these product measures using optimal transport maps $T$ that push forward a reference measure (typically a standard Gaussian $\rho$) to the target. By "lifting" the problem into this map space, the previously non-convex optimization is transformed into a convex optimization problem over a Hilbert space—specifically, a Gaussian Sobolev space.
This shift in perspective allows for the application of rigorous functional analysis tools. The authors utilize first-order optimality conditions to derive stability bounds and employ the Lax-Milgram theorem to prove that the optimizer is Fréchet differentiable. Furthermore, they demonstrate that the derivative of the MFVI optimizer can be characterized as the solution to a specific partial differential equation (PDE), providing a precise mechanism for sensitivity analysis.
Key Theoretical Insights
Dimension-Free Stability
One of the most significant findings is that the stability of MFVI does not necessarily degrade as the dimensionality $d$ of the problem increases. The authors establish a global Lipschitz bound:
$$W_2(\tilde{\nu}^*, \nu^*) \leq \frac{\|\nabla V - \nabla \tilde{V}\|_{L^2(\tilde{\nu}^*)}}{\alpha}$$
In this expression, the distance between two MFVI optimizers (measured in the 2-Wasserstein metric) is bounded by the difference in the gradients of their potentials, scaled inversely by the strong log-concavity parameter $\alpha$. Crucially, this constant is independent of the dimension, suggesting that MFVI remains robust even in very high-dimensional settings.
Quantitative Bernstein–von Mises Theorem
The paper provides a quantitative Bernstein–von Mises (BvM) theorem for MFVI. This result bridges the gap between variational inference and classical asymptotic theory, proving that as the sample size grows, the MFVI posterior converges to a specific Gaussian approximation at a measurable rate. This offers a theoretical "safety net" for using VI in large-scale data regimes.
Applications and Implications
- Robust Bayesian Inference: The stability results ensure that small modeling errors or outliers do not lead to disproportionately large errors in the variational posterior, a vital property for reliable automated inference.
- Empirical Bayes: The differentiability of the MFVI optimizer allows for the use of gradient-based methods to learn model hyperparameters directly from data, as the sensitivity of the posterior to these parameters is now well-defined.
- Distributed Stochastic Control: The framework extends to decentralized systems where multiple agents must act based on local information, providing stability guarantees for the resulting control laws.
Limitations and Future Directions
While this work provides a foundational theory for MFVI stability, it is currently constrained to strongly log-concave target distributions. This class includes many important models, such as Gaussian processes and logistic regression, but excludes multimodal distributions or those with heavy tails. Future research may explore extending these "lifting" techniques to Structured Variational Inference, where dependencies between variables are partially preserved, or developing efficient computational methods to solve the PDEs that characterize the optimizer's sensitivity in practice.
Summary updated