Abstract
Bayesian inference plays an important role in advancing machine learning, but faces computational challenges when applied to complex models such as deep neural networks. Variational inference circumvents these challenges by formulating Bayesian inference as an optimization problem and solving it using gradient-based optimization. In this paper, we argue in favor of natural-gradient approaches which, unlike their gradient-based counterparts, can improve convergence by exploiting the information geometry of the solutions. We show how to derive fast yet simple natural-gradient updates by using a duality associated with exponential-family distributions. An attractive feature of these methods is that, by using natural-gradients, they are able to extract accurate local approximations for individual model components. We summarize recent results for Bayesian deep learning showing the superiority of natural-gradient approaches over their gradient counterparts.
The Challenge of Euclidean Optimization in Bayesian Inference
Bayesian inference is highly desirable in fields like medical diagnostics and robotics because it provides a measure of uncertainty alongside its predictions. However, calculating exact Bayesian posteriors is computationally intractable for large models like deep neural networks.
Variational Inference (VI) addresses this by approximating the complex posterior with a simpler distribution, turning inference into an optimization problem. Typically, this is optimized using Stochastic Gradient Descent (SGD). However, SGD optimizes in Euclidean space, which is often an inappropriate metric for probability distributions. As illustrated below, two pairs of Gaussian distributions can have the same Euclidean distance between their parameters while having vastly different levels of overlap, meaning the "distance" in terms of information is not captured by Euclidean metrics.
Natural-gradient descent fixes this by using the Fisher Information Matrix (FIM) to navigate the Riemannian space of the distributions. Historically, computing and inverting the FIM has been prohibitively expensive, limiting natural gradients to simple, "conditionally-conjugate" models. This paper introduces a method to bring the benefits of natural gradients to complex, nonconjugate models like Bayesian Neural Networks (BNNs).
Leveraging Duality for Efficient Natural Gradients
The authors bypass the computational bottleneck of the FIM by exploiting the duality of exponential-family distributions. These distributions can be parameterized by natural parameters ($\lambda$) or expectation parameters ($\mu$). Because these spaces are related through a Legendre transform, they are duals of each other.
A key theoretical contribution of this work is the proof that the natural gradient with respect to $\lambda$ is exactly equal to the standard gradient with respect to $\mu$. This allows for the computation of natural gradients without explicitly calculating or inverting the FIM. For complex models, taking the gradient of likelihood terms with respect to expectation parameters yields a simple stochastic natural-gradient update.
For practical deep learning applications, the authors highlight the Variational Online Gauss-Newton (VOGN) approximation. VOGN utilizes a diagonal covariance matrix and a generalized Gauss-Newton approximation for the Hessian, making the natural-gradient update nearly as simple to implement as standard optimizers like Adam.
Empirical Performance on Benchmark Datasets
The proposed approximate natural-gradient method (VOGN) was evaluated against a standard gradient-based VI method, Bayes by Backprop (using the Adam optimizer), on two benchmark datasets: the Australian dataset and the Breast Cancer dataset.
The results demonstrate significant performance gains:
- Faster Convergence: On both datasets, the natural-gradient method converged drastically faster than its Euclidean counterpart. VOGN reached near-optimal test $\log_2$ loss in under 200 epochs.
- Superior Optima: The standard gradient-based method was still descending at 2,000 epochs and failed to reach the low loss values achieved rapidly by the natural-gradient approach.
Conclusion and Future Directions
While the natural-gradient approach offers substantial speedups, it currently relies on approximations like diagonal covariance matrices to remain computationally feasible for deep neural networks. Future work involves leveraging modern automatic-differentiation tools to compute these gradients for a wider variety of models and exploring stochastic approximation methods for models like Gaussian processes and linear dynamical systems. By respecting the underlying geometry of probability distributions, this work provides a path toward more efficient and robust Bayesian deep learning.
Summary updated