Provable Smoothness Guarantees for Black-Box Variational Inference

Justin Domke

Introduction

Black-box variational inference (BBVI) is a foundational technique in modern Bayesian statistics, used to approximate complex probability distributions through gradient-based optimization. This paper provides the first rigorous proof that for location-scale variational families, a smooth target distribution results in a smooth "energy" term in the variational objective. By establishing these provable guarantees, the author demonstrates how to design robust optimization algorithms—such as proximal gradient descent—that avoid the erratic instability common in standard BBVI.

The Challenge of Non-Smoothness in BBVI

BBVI works by minimizing the Evidence Lower Bound (ELBO), which balances an energy term (how well the approximation fits the data) and an entropy term (how spread out the approximation is).

To guarantee that an optimization algorithm will reliably find the best approximation, theorists typically require the objective function to be "smooth" (meaning its gradients do not change abruptly). While many researchers have historically assumed the BBVI objective was smooth, it remained unproven. Furthermore, the entropy term becomes notoriously non-smooth when the variance of the approximation shrinks. This often leads to unstable optimization where the algorithm "jumps" away from the optimal solution.

Gradient descent on the 'fires' dataset shows extreme sensitivity to initialization and step-size, leading to worrying 'jumps' in the objective.

Methodology: Location-Scale Families and Smoothness

The study focuses on location-scale families, a broad class of distributions (including Gaussians and Student-T) defined by a mean (location) and a covariance matrix (scale).

  1. Provable Smoothness: Using a specific inner-product space and Bessel’s inequality, the author proves that if the target log-density is $M$-smooth, the energy component of the ELBO inherits this $M$-smoothness.
  2. Solution Guarantees: The paper proves that the optimal variational distribution cannot have an arbitrarily small variance if the target is smooth. This establishes a "safe," high-variance region in the parameter space.
  3. Robust Optimization: To handle the non-smooth entropy term, the paper proposes two strategies:
  4. Projected Gradient Descent: Restricting the parameters to remain within the "safe" high-variance region.
  5. Proximal Gradient Descent: Linearizing the smooth energy term while treating the non-smooth entropy term exactly.

Experimental Validation and Key Insights

The theoretical claims are evaluated using linear regression (boston, fires) and binary logistic regression (australian, ionosphere). These models allow for exact ELBO and gradient computation, isolating optimization surface effects from stochastic noise.

  • Smoothness is Inherited: The most significant theoretical breakthrough (Theorem 1) is that the smoothness of the target distribution is preserved through the expectation operation. If the model is well-behaved, the variational energy objective will be too.
  • The Entropy Trap: The failure of "naive" gradient descent in BBVI is driven by entropy, not energy. As the scale (variance) approaches zero, the entropy gradient explodes.
  • Scale Sensitivity: Naive optimization performs similarly to proximal methods at large initial scales but fails catastrophically at small scales. This confirms that BBVI's non-smoothness is primarily a small-scale phenomenon.
Results showing that naive optimization is similar to proximal for large initial scales but fails significantly for small scales.

Ultimately, proximal gradient descent effectively "tames" the entropy term. It allows the algorithm to be initialized stably even at zero variance, avoiding the catastrophic jumps of standard methods.

Limitations and Future Directions

  • Exact vs. Stochastic Gradients: The current theory assumes exact gradients are available. In practice, BBVI uses stochastic gradient estimates. While smoothness helps control gradient variance, a complete convergence proof for practical BBVI requires integrating stochastic approximation theory.
  • Family Constraints: The proofs apply specifically to location-scale families. Extending these guarantees to more complex variational families, such as Normalizing Flows, remains an open challenge.
  • Computational Overhead: While the proximal operator for the entropy term is highly efficient ($O(d)$), the projection operator requires a singular value decomposition, making it computationally heavier ($O(d^3)$).

Summary updated

Abstract

Black-box variational inference tries to approximate a complex target distribution through a gradient-based optimization of the parameters of a simpler distribution. Provable convergence guarantees...
BibTeX
@inproceedings{domke2020provable,
  author = {Justin Domke},
  title = {{Provable Smoothness Guarantees for Black-Box Variational Inference}},
  booktitle = {International Conference on Machine Learning},
  year = {2020},
  url = {https://openreview.net/forum?id=HbwCV6eSNBr},
}
Figure