Provable Smoothness Guarantees for Black-Box Variational Inference

Justin Domke

Abstract: Black-box variational inference tries to approximate a complex target distribution though a gradient-based optimization of the parameters of a simpler distribution. Provable convergence guarantees require structural properties of the objective. This paper shows that for location-scale family approximations, if the target is M-Lipschitz smooth, then so is the objective, if the entropy is excluded. The key proof idea is to describe gradients in a certain inner-product space, thus permitting use of Bessel's inequality. This result gives insight into how to parameterize distributions, gives bounds the location of the optimal parameters, and is a key ingredient for convergence guarantees.

Introduction

Variational Inference (VI) is a cornerstone of modern Bayesian statistics, transforming difficult integration tasks into optimization problems. "Black-Box" VI (BBVI) is particularly popular because it only requires the gradient of the target log-density, making it applicable to a wide range of models. However, a significant theoretical gap has persisted: most optimization convergence proofs require the objective function to be Lipschitz smooth (meaning the gradient doesn't change too abruptly).

While researchers often assumed the BBVI objective was smooth, practitioners frequently observe "jumps" or instabilities during optimization that remained poorly understood. Without a formal proof of smoothness, the convergence of BBVI remained theoretically uncertain.

Instability of naive gradient descent in BBVI across different initializations and step sizes. The plots show how suboptimality can spike or diverge depending on the initial covariance $C_0$ and step size.

Theoretical Foundations

The paper focuses on location-scale families (such as Gaussian or Student-t distributions) where the variational distribution is parameterized by a mean $m$ and a covariance-related matrix $C$. The primary technical contribution is a proof that smoothness is "inherited" from the target model.

By describing gradients in a specific inner-product space over functions and applying Bessel’s inequality, the author proves that if the target log-density is $M$-smooth, the energy component of the Evidence Lower Bound (ELBO) is also $M$-smooth. This provides the first rigorous foundation for the smoothness assumptions used in BBVI.

Addressing the Entropy Term

While the energy term is smooth, the Entropy term is not. As the variance of the variational distribution approaches zero, the gradient of the entropy explodes, causing the instabilities seen in Figure 1. To handle this, the paper proposes two robust algorithms:

  • Projected Gradient Descent: This method constrains optimization to a region where variance is large enough to keep the entropy term well-behaved.
  • Proximal Gradient Descent: This approach uses a "proximal operator" to handle the entropy term analytically while linearizing the energy term.

Experimental Evaluation

The author evaluates these methods on standard regression and classification datasets: boston (housing), fires (forest fires), australian (credit applications), and ionosphere (radar data). These datasets allow for exact (or near-exact) gradient computation, isolating the optimization algorithm's performance from stochastic noise.

Convergence of projected, proximal, and naive gradient descent across various datasets and scaling factors. The proximal (prox) and projected (proj) methods show superior stability and convergence compared to the naive approach, especially at extreme scaling values.

As shown in Figure 2, naive gradient descent is highly sensitive to initialization and scaling. If initialized with small variance, it often fails or exhibits wild jumps. In contrast, the proposed proximal and projected methods converge reliably across all tested scenarios.

Key Insights and Conclusion

  • Smoothness is Inherited: The smoothness of the target model directly translates to the smoothness of the variational energy. If the model is well-behaved, the objective is too.
  • The Entropy Culprit: The negative entropy term is the primary source of instability in BBVI. The author proves that at the optimal solution, the eigenvalues of the covariance matrix must be at least $1/M$, meaning the optimal approximation cannot be "too skinny."
  • Algorithm Superiority: Proximal gradient descent is generally superior to projected methods because it is computationally faster and offers a better convergence rate ($M$ vs $2M$).

While these results rely on the target log-density being Lipschitz smooth—which excludes models with hard constraints—they provide a vital theoretical framework for understanding and improving BBVI in practice.

Summary updated

Abstract

Black-box variational inference tries to approximate a complex target distribution though a gradient-based optimization of the parameters of a simpler distribution. Provable convergence guarantees require structural properties of the objective. This paper shows that for location-scale family approximations, if the target is M-Lipschitz smooth, then so is the objective, if the entropy is excluded. The key proof idea is to describe gradients in a certain inner-product space, thus permitting use of Bessel's inequality. This result gives insight into how to parameterize distributions, gives bounds the location of the optimal parameters, and is a key ingredient for convergence guarantees.
BibTeX
@inproceedings{domke2020provable,
  author = {Justin Domke},
  title = {{Provable Smoothness Guarantees for Black-Box Variational Inference}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=HbwCV6eSNBr},
}

Loading related content...

Figure