Abstract
We investigate the use of alternative divergences to Kullback-Leibler (KL) in variational inference (VI), based on the Variational Dropout framework. Stochastic gradient variational Bayes (SGVB) is a general framework for estimating the evidence lower bound (ELBO) in Variational Bayes. In this work, we extend the SGVB estimator using Alpha-Divergences, which are alternatives to the KL divergence objective in VI. Gaussian dropout can be seen as a local reparametrization trick of the SGVB objective. We extend Variational Dropout to use alpha divergences for variational inference. Our results compare $\alpha$-divergence variational dropout with standard variational dropout using correlated and uncorrelated weight noise. We show that the $\alpha$-divergence with $\alpha \rightarrow 1$ (or KL divergence) remains a robust measure for variational inference, yielding the lowest training error and optimizing a strong lower bound for the ELBO among all values of the parameter $\alpha \in [0,\infty)$.
Introduction
In Bayesian Deep Learning, Variational Inference (VI) is typically employed to approximate the posterior distribution of network weights by minimizing the Kullback-Leibler (KL) divergence between the approximation and the true posterior. This process maximizes the Evidence Lower Bound (ELBO). However, KL divergence is known to underestimate uncertainty. Variational Dropout interprets dropout as a form of approximate Bayesian inference. This study explores whether generalizing the objective function to the broader family of $\alpha$-divergences—including KL, Hellinger, and $\chi^2$—can improve regularization or predictive accuracy in neural networks.
Methods
The researchers extended the Stochastic Gradient Variational Bayes (SGVB) estimator to incorporate $\alpha$-divergences ($D_\alpha$) instead of the standard KL term. The methodology involved:
- Generalizing the ELBO: Replacing the KL term with $D_\alpha$.
- Weight Decomposition: Decomposing weights into a deterministic magnitude and a random sign/scale variable with a log-uniform prior.
- Numerical Approximation: Using Monte Carlo integration for intractable divergence terms and fitting a 3rd-order interpolating polynomial to pre-computed values to ensure efficient gradient computation during training.
- Architectural Variants: Testing "Variational A" (correlated weight noise) and "Variational B" (independent weight noise) on the MNIST dataset across multiple runs to ensure stability.
Results and Discussion
The Dominance of KL Divergence
Empirical results indicate that despite the flexibility offered by the $\alpha$ parameter, performance peaks as $\alpha$ approaches 1. This suggests that the KL divergence is uniquely suited for the geometry of the Variational Dropout objective. As shown in the figure below, the average test set error reaches a minimum near $\alpha = 1$ across various hidden layer sizes, regardless of the model capacity.
Impact of Noise Correlation
The choice of $\alpha$ significantly impacts models with correlated weight noise (where noise depends on the weight value). In contrast, for models with independent (uncorrelated) noise, the value of $\alpha$ has a negligible effect on final test accuracy. The following comparison illustrates this disparity: in the correlated case (a), different $\alpha$ values lead to distinct performance trajectories, while in the uncorrelated case (b), the curves are tightly clustered, indicating that the regularization benefit of specific $\alpha$ values is tied to the noise structure.
Conclusion
While $\alpha$-divergences provide a theoretically rich framework for Variational Dropout, the standard KL divergence ($\alpha \to 1$) remains the most effective choice for minimizing test error on MNIST. Future work should investigate these findings on more complex datasets like CIFAR-10 and within different architectures such as Convolutional Neural Networks. Additionally, exploring automated tuning of $\alpha$ as a learned parameter could offer further insights into optimal divergence metrics for Bayesian inference.
Summary updated