Abstract
This paper introduces the variational Rényi bound (VR) that extends traditional variational inference to Rényi's alpha-divergences. This new family of variational methods unifies a number of existing approaches and enables a smooth interpolation from the evidence lower-bound to the log (marginal) likelihood, controlled by the value of alpha. The reparameterization trick, Monte Carlo approximation, and stochastic optimization methods are deployed to obtain a tractable and unified framework. We further consider negative alpha values and propose a novel variational inference method, VR-max. Experiments on Bayesian neural networks and variational auto-encoders demonstrate the wide applicability of the VR bound.
Introduction: The Limitations of Traditional VI
In probabilistic machine learning, Variational Inference (VI) is a cornerstone technique for approximating complex posterior distributions. However, traditional VI—which relies on the Kullback-Leibler (KL) divergence—is often criticized for being "over-confident." It tends to fit a single mode of a distribution while ignoring others, a property known as "zero-forcing."
Li and Turner (2016) address this by introducing a powerful generalization called Rényi Divergence Variational Inference. By swapping the standard KL divergence for the broader family of Rényi $\alpha$-divergences, they provide a unified framework that connects several state-of-the-art inference methods and offers a tunable knob to balance approximation quality.
The Variational Rényi (VR) Bound
The core contribution is the Variational Rényi (VR) bound, a new family of objective functions $L_\alpha$. This bound is proven to be continuous and non-increasing with respect to the parameter $\alpha$. By adjusting $\alpha$, researchers can smoothly interpolate between different types of approximations:
- $\alpha \to 1$: Recovers standard Variational Inference (KL divergence).
- $\alpha = 0$: Recovers the Importance Weighted Auto-encoder (IWAE) objective.
- Negative $\alpha$: Provides even tighter bounds, leading to a new algorithm called VR-max (where $\alpha \to -\infty$).
As shown in the figure above, changing $\alpha$ allows the model to move from mode-seeking behavior (fitting one peak tightly at high $\alpha$) to mass-covering behavior (spreading the approximation to cover the full distribution at lower $\alpha$).
Unifying the Inference Landscape
One of the most significant insights of this work is the theoretical "bridge" it builds between previously siloed methods. The authors demonstrate that Stochastic Expectation Propagation (SEP) and Black-Box Alpha (BB-$\alpha$) are specific stochastic approximations of the VR bound.
This unification allows practitioners to choose the best type of approximation for a given task within a single framework. For instance, the authors find that $\alpha=0.5$ (related to the Hellinger distance) often performs best as a symmetric middle ground between the extremes of VI and EP.
Experimental Evaluation
The framework was evaluated across two primary domains:
- Regression (Bayesian Neural Networks): Tested on 10 UCI datasets (e.g., Boston Housing, Concrete). Results showed that mass-covering regimes (lower $\alpha$) generally yielded better test log-likelihoods and better-calibrated uncertainty.
- Generative Modeling (VAEs): Evaluated on image datasets including MNIST, OMNIGLOT, and Frey Face. The proposed VR-max algorithm performed comparably to or better than the then-state-of-the-art IWAE. Notably, VR-max can be up to 3x faster than IWAE because it only requires backpropagation through the "best" sample rather than a weighted average.
Conclusion and Future Directions
The VR bound provides a flexible and theoretically grounded framework for approximate inference. While it offers a powerful "tunable knob" via $\alpha$, choosing the optimal value currently requires cross-validation. Future work may explore automatically adapting $\alpha$ during training or further investigating the bias-variance trade-offs inherent in Monte Carlo approximations of these bounds, particularly for negative $\alpha$ values.
Reference: Li, Y., & Turner, R. E. (2016). Rényi Divergence Variational Inference. Advances in Neural Information Processing Systems (NIPS).
Summary updated