Approximation Based Variance Reduction for Reparameterization Gradients

Tomas Geffner, Justin Domke

TL;DR

This paper introduces a novel control variate technique to significantly reduce the variance of gradient estimators used in variational inference. This method allows for more efficient optimization of complex, non-factorized variational distributions, leading to faster and more reliable training of probabilistic models.

What Problem Does This Paper Address?

Variational Inference (VI) is a powerful technique for approximating intractable posterior distributions in Bayesian models. A key challenge in VI is estimating the gradients of the objective function (often the Evidence Lower Bound, or ELBO) with respect to the parameters of the variational distribution. The "reparameterization trick" is a popular method for obtaining low-variance gradient estimators, but its effectiveness can be limited, especially when using more flexible, non-factorized variational distributions (like Gaussians with arbitrary covariance structures). These distributions can capture complex dependencies between variables but lead to higher variance in gradient estimates, making optimization difficult and slow. Existing variance reduction techniques, like Taylor-based control variates, struggle with these more complex distributions, particularly for the covariance parameters.

Methods

The core contribution is a new control variate designed to work with any reparameterizable distribution that has a known mean and covariance matrix. The method's key ideas are:

  • Quadratic Approximation: The control variate is based on a quadratic approximation of the model's log-probability function. This approximation, denoted $\hat{f}$, is parameterized by a set of coefficients $v$.
  • Tractable Control Variate: The control variate itself is defined as the difference between the exact gradient of the expected log-probability and the gradient of the approximated log-probability. Crucially, this control variate is shown to be tractable for distributions like Gaussians (with any covariance structure), provided their mean and covariance are known.
  • Double-Descent Optimization: The parameters $v$ of the quadratic approximation are learned simultaneously with the variational parameters $w$. This is achieved through a "double-descent" scheme that aims to minimize the variance of the final gradient estimator. Two methods are proposed for minimizing this variance: one directly minimizing the variance of the final estimator, and another using a proxy objective that minimizes the difference between the true gradient and the approximated gradient.

Datasets

The paper uses the following datasets for evaluation:

  • ala dataset: For Bayesian logistic regression.
  • frisk dataset: For hierarchical Poisson models.
  • Red-wine dataset: For Bayesian neural networks.

Key Insights and Ideas

  • Beyond Taylor: The paper demonstrates that simple Taylor approximations, while useful for basic cases, are insufficient for capturing the variance contributions from covariance parameters in more complex distributions. Their proposed quadratic approximation, optimized via double-descent, offers a more robust solution.
  • Covariance Matters: For non-factorized distributions, the variance in gradient estimates is often dominated by the covariance parameters. This is where traditional Taylor-based methods fail, and where the proposed control variate shows significant improvements.
  • Double-Descent Synergy: Learning the control variate parameters ($v$) concurrently with the variational parameters ($w$) is crucial. This allows the control variate to adapt as the optimization progresses, leading to sustained variance reduction.
  • Efficiency Gains: The proposed method, despite adding a control variate, has a manageable computational overhead, making it practical for real-world applications.

Key Results

  • Orders of Magnitude Variance Reduction: The proposed control variate achieves variance reductions of "several orders of magnitude" compared to Taylor-based methods, especially for diagonal-plus-low-rank and full-rank Gaussian variational distributions.
  • Improved Optimization: This variance reduction translates directly to significantly improved optimization convergence speed and reliability.
  • Competitive with Fewer Samples: In some experiments, the proposed method with fewer samples (M=10) outperformed competing approaches using more samples (M=50).
  • Robustness to Step Size: The method is shown to be less sensitive to the choice of learning rate, leading to more stable training.
  • Manageable Overhead: The computational overhead of the proposed control variate is around 50%, compared to approximately 150% for Taylor-based methods.

Limitations and Future Work

  • Tractability of Approximations: While the control variate is tractable for Gaussians with known mean and covariance, extending it to distributions where these are not easily computed (e.g., Householder flows) requires more complex derivations.
  • Approximation Quality: The effectiveness of the control variate relies on the quality of the quadratic approximation. While the double-descent scheme helps, further research could explore more sophisticated approximation strategies.
  • Future Work: The authors suggest that their control variate can be combined with other variance reduction techniques (like the sticking-the-landing estimator) and applied to other gradient estimators beyond reparameterization. They also highlight the potential for this method to be integrated into automated VI tools.

Summary updated

Abstract

Flexible variational distributions improve variational inference but are harder to optimize. In this work we present a control variate that is applicable for any reparameterizable distribution with known mean and covariance matrix, e.g. Gaussians with any covariance structure. The control variate is based on a quadratic approximation of the model, and its parameters are set using a double-descent scheme by minimizing the gradient estimator's variance. We empirically show that this control variate leads to large improvements in gradient variance and optimization convergence for inference with non-factorized variational distributions.
BibTeX
@inproceedings{geffner2020approximation,
  author = {Tomas Geffner and Justin Domke},
  title = {{Approximation Based Variance Reduction for Reparameterization Gradients}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=SFUWrtcmBg9},
}

Loading related content...

Figure