Alpha-Divergences in Variational Dropout

Bogdan Mazoure, Riashat IslamarXiv 2017 · OpenReview

Abstract

We investigate the use of alternative divergences to Kullback-Leibler (KL) in variational inference (VI), based on the Variational Dropout framework. Stochastic gradient variational Bayes (SGVB) is a general framework for estimating the evidence lower bound (ELBO) in Variational Bayes. In this work, we extend the SGVB estimator using Alpha-Divergences, which are alternatives to the KL divergence objective in VI. Gaussian dropout can be seen as a local reparametrization trick of the SGVB objective. We extend Variational Dropout to use alpha divergences for variational inference. Our results compare $\alpha$-divergence variational dropout with standard variational dropout using correlated and uncorrelated weight noise. We show that the $\alpha$-divergence with $\alpha \rightarrow 1$ (or KL divergence) remains a robust measure for variational inference, yielding the lowest training error and optimizing a strong lower bound for the ELBO among all values of the parameter $\alpha \in [0,\infty)$.

Introduction

In Bayesian Deep Learning, Variational Inference (VI) is typically employed to approximate the posterior distribution of network weights by minimizing the Kullback-Leibler (KL) divergence between the approximation and the true posterior. This process maximizes the Evidence Lower Bound (ELBO). However, KL divergence is known to underestimate uncertainty. Variational Dropout interprets dropout as a form of approximate Bayesian inference. This study explores whether generalizing the objective function to the broader family of $\alpha$-divergences—including KL, Hellinger, and $\chi^2$—can improve regularization or predictive accuracy in neural networks.

Methods

The researchers extended the Stochastic Gradient Variational Bayes (SGVB) estimator to incorporate $\alpha$-divergences ($D_\alpha$) instead of the standard KL term. The methodology involved:

  1. Generalizing the ELBO: Replacing the KL term with $D_\alpha$.
  2. Weight Decomposition: Decomposing weights into a deterministic magnitude and a random sign/scale variable with a log-uniform prior.
  3. Numerical Approximation: Using Monte Carlo integration for intractable divergence terms and fitting a 3rd-order interpolating polynomial to pre-computed values to ensure efficient gradient computation during training.
  4. Architectural Variants: Testing "Variational A" (correlated weight noise) and "Variational B" (independent weight noise) on the MNIST dataset across multiple runs to ensure stability.

Results and Discussion

The Dominance of KL Divergence

Empirical results indicate that despite the flexibility offered by the $\alpha$ parameter, performance peaks as $\alpha$ approaches 1. This suggests that the KL divergence is uniquely suited for the geometry of the Variational Dropout objective. As shown in the figure below, the average test set error reaches a minimum near $\alpha = 1$ across various hidden layer sizes, regardless of the model capacity.

Average test set error across different alpha values and hidden layer sizes. The minimum error is consistently observed near alpha = 1.

Impact of Noise Correlation

The choice of $\alpha$ significantly impacts models with correlated weight noise (where noise depends on the weight value). In contrast, for models with independent (uncorrelated) noise, the value of $\alpha$ has a negligible effect on final test accuracy. The following comparison illustrates this disparity: in the correlated case (a), different $\alpha$ values lead to distinct performance trajectories, while in the uncorrelated case (b), the curves are tightly clustered, indicating that the regularization benefit of specific $\alpha$ values is tied to the noise structure.

Comparison of test set error for correlated (a) and uncorrelated (b) noise dropout. Alpha has a much larger impact when weight noise is correlated.

Conclusion

While $\alpha$-divergences provide a theoretically rich framework for Variational Dropout, the standard KL divergence ($\alpha \to 1$) remains the most effective choice for minimizing test error on MNIST. Future work should investigate these findings on more complex datasets like CIFAR-10 and within different architectures such as Convolutional Neural Networks. Additionally, exploring automated tuning of $\alpha$ as a learned parameter could offer further insights into optimal divergence metrics for Bayesian inference.

Summary updated

Abstract

We investigate the use of alternative divergences to Kullback-Leibler (KL) in variational inference(VI), based on the Variational Dropout \cite{kingma2015}. Stochastic gradient variational Bayes (SGVB) \cite{aevb} is a general framework for estimating the evidence lower bound (ELBO) in Variational Bayes. In this work, we extend the SGVB estimator with using Alpha-Divergences, which are alternative to divergences to VI' KL objective. The Gaussian dropout can be seen as a local reparametrization trick of the SGVB objective. We extend the Variational Dropout to use alpha divergences for variational inference. Our results compare $\alpha$-divergence variational dropout with standard variational dropout with correlated and uncorrelated weight noise. We show that the $\alpha$-divergence with $\alpha \rightarrow 1$ (or KL divergence) is still a good measure for use in variational inference, in spite of the efficient use of Alpha-divergences for Dropout VI \cite{Li17}. $\alpha \rightarrow 1$ can yield the lowest training error, and optimizes a good lower bound for the evidence lower bound (ELBO) among all values of the parameter $\alpha \in [0,\infty)$.
BibTeX
@misc{mazoure2017alpha,
  author = {Bogdan Mazoure and Riashat Islam},
  title = {{Alpha-Divergences in Variational Dropout}},
  year = {2017},
  eprint = {1711.04345},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1711.04345},
}
Figure