Practical and Matching Gradient Variance Bounds for Black-Box Variational Bayesian Inference

Kyurae Kim, Kaiwen Wu, Jisu Oh, Jacob R. Gardner

Abstract

Understanding the gradient variance of black-box variational inference (BBVI) is a crucial step for establishing its convergence and developing algorithmic improvements. However, existing studies have yet to show that the gradient variance of BBVI satisfies the conditions used to study the convergence of stochastic gradient descent (SGD), the workhorse of BBVI. In this work, we show that BBVI satisfies a matching bound corresponding to the $ABC$ condition used in the SGD literature when applied to smooth and quadratically-growing log-likelihoods. Our results generalize to nonlinear covariance parameterizations widely used in the practice of BBVI. Furthermore, we show that the variance of the mean-field parameterization has provably superior dimensional dependence.

Introduction

Black-Box Variational Inference (BBVI) has become a cornerstone of modern Bayesian machine learning, enabling scalable inference in complex probabilistic models. By utilizing Stochastic Gradient Descent (SGD) to maximize the Evidence Lower Bound (ELBO), BBVI powers popular probabilistic programming libraries such as Stan, Pyro, and Turing. Despite its widespread adoption, a theoretical gap has persisted: the convergence guarantees for SGD rely on specific assumptions regarding the variance of the gradient estimator.

Until recently, it was unclear whether the gradient estimators used in BBVI—particularly those employing the "reparameterization trick" and nonlinear "bijectors" (like the softplus function) to ensure positive scale parameters—actually satisfied these theoretical conditions. Without this link, practitioners could not strictly guarantee convergence without resorting to heuristic modifications, such as artificial gradient clipping or bounding the search space. This paper addresses this gap by proving that BBVI gradient estimators satisfy the necessary conditions for convergence under realistic settings.

Theoretical Framework: The ABC Condition

The authors focus on the location-scale variational family (e.g., Gaussians) and investigate gradient variance behavior under smooth, quadratically-growing log-likelihoods. The central contribution is the proof that BBVI gradients satisfy the "Expected Smoothness" or ABC condition:

$$\mathbb{E}[\|g\|^2] \leq 2A(F(\lambda) - F^*) + B\|\nabla F(\lambda)\|^2 + C$$

This condition is a modern, relaxed assumption in SGD theory that is significantly more realistic than the traditional "bounded variance" assumption. By establishing that BBVI satisfies this bound, the authors enable the application of existing SGD convergence proofs, demonstrating that BBVI converges to a neighborhood of the optimum at a rate of $\mathcal{O}(1/T)$ for convex problems.

Dimensionality and Parameterization

A key insight of this research is the impact of covariance parameterization on gradient variance. The authors provide matching upper and lower bounds that reveal a stark difference in how variance scales with dimensionality ($d$):

  • Mean-Field Parameterization: The gradient variance scales as $O(\sqrt{d})$.
  • Full-Rank Parameterization: The gradient variance scales as $O(d)$.

This provides a rigorous theoretical justification for the empirical observation that mean-field approximations are often more stable and easier to optimize in high-dimensional settings. Furthermore, the analysis accounts for nonlinear "conditioners" used to maintain positive definiteness in scale matrices. Interestingly, the results suggest that using a nonlinear conditioner like softplus can actually lead to lower variance compared to simpler linear matrix square root parameterizations.

Empirical Validation

The theoretical bounds were validated using both synthetic quadratic problems and real-world benchmarks from the UCI Machine Learning Repository, including the FERTILITY, PENDULUM, AIRFOIL, and WINE datasets.

In experiments involving linear regression via BBVI, the theoretical upper bounds closely tracked the actual observed gradient variance throughout the optimization process. Empirical results confirmed the predicted scaling laws, with mean-field parameterizations consistently exhibiting lower variance than full-rank Cholesky or matrix square root methods. This alignment between theory and practice suggests that the standard reparameterization trick used in libraries like PyTorch and JAX naturally satisfies the conditions required for stable convergence without the need for ad-hoc "hacks."

Discussion and Limitations

While this work provides a significant step forward in the theoretical understanding of BBVI, several limitations remain for future exploration:

  1. Likelihood Constraints: The proofs currently apply to log-likelihoods that are smooth and quadratically growing, excluding models with heavy tails or non-smooth components.
  2. Family Specificity: The results are tailored to the location-scale family used in Automatic Differentiation Variational Inference (ADVI).
  3. Conditioner Growth: The theory requires "1-Lipschitz" conditioners (e.g., softplus). It does not yet extend to the exponential function, which grows too rapidly for the current proof techniques.
  4. Mean-Field Tightness: Although the scaling factor is correct, the authors note that the mean-field upper bounds remain empirically loose compared to actual variance, suggesting that tighter constants may yet be discovered.

By providing matching lower bounds, the authors demonstrate that the ABC condition is essentially the most fundamental assumption that can describe BBVI's behavior for this class of functions, marking a definitive milestone in the study of variational gradient estimators.

Summary updated

Abstract

Understanding the gradient variance of black-box variational inference (BBVI) is a crucial step for establishing its convergence and developing algorithmic improvements. However, existing studies have yet to show that the gradient variance of BBVI satisfies the conditions used to study the convergence of stochastic gradient descent (SGD), the workhorse of BBVI. In this work, we show that BBVI satisfies a matching bound corresponding to the $ABC$ condition used in the SGD literature when applied to smooth and quadratically-growing log-likelihoods. Our results generalize to nonlinear covariance parameterizations widely used in the practice of BBVI. Furthermore, we show that the variance of the mean-field parameterization has provably superior dimensional dependence.
BibTeX
@inproceedings{kim2023practical,
  author = {Kyurae Kim and Kaiwen Wu and Jisu Oh and Jacob R. Gardner},
  title = {{Practical and Matching Gradient Variance Bounds for Black-Box Variational Bayesian Inference}},
  booktitle = {International Conference on Machine Learning},
  year = {2023},
  url = {https://openreview.net/forum?id=VXIRjBCV4Z},
}
Figure