Abstract
The natural gradient method has been used effectively in conjugate Gaussian process models, but the non-conjugate case has been largely unexplored. We examine how natural gradients can be used in non-conjugate stochastic settings, together with hyperparameter learning. We conclude that the natural gradient can significantly improve performance in terms of wall-clock time. For ill-conditioned posteriors the benefit of the natural gradient method is especially pronounced, and we demonstrate a practical setting where ordinary gradients are unusable. We show how natural gradients can be computed efficiently and automatically in any parameterization, using automatic differentiation. Our code is integrated into the GPflow package.
Introduction: The Geometry of Variational Inference
Variational Inference (VI) is a cornerstone of modern Bayesian machine learning, allowing us to approximate complex posterior distributions with simpler, tractable ones. In Gaussian Process (GP) models, this typically involves optimizing the parameters of a Gaussian approximation. Standard optimization techniques, such as Gradient Descent (GD) or Adam, treat these parameters as coordinates in a Euclidean space. However, this perspective is "unnatural" because it ignores the information-geometric structure of the space of probability distributions.
A small change in the parameters of a narrow Gaussian can represent a much larger change in the distribution than the same change in a wide Gaussian. This mismatch leads to two primary issues: extreme sensitivity to how the model is parameterized and poor performance in "ill-conditioned" scenarios where the optimization landscape is highly distorted. Natural Gradient Descent (NGD) resolves these issues by scaling the gradient by the Fisher Information Matrix, ensuring that optimization steps are consistent with the underlying geometry of the distributions.
Methods: Automating Natural Gradients
The authors extend NGD to non-conjugate GP models, where it has historically been difficult to apply. They introduce an efficient "Forward-Mode Trick" that involves running reverse-mode automatic differentiation twice to simulate forward-mode differentiation, allowing natural gradients to be computed automatically for any parameterization. This removes the need for tedious manual derivations and allows for seamless integration into modern software frameworks like GPflow.
Because natural gradients apply specifically to the variational distribution, the authors propose a hybrid optimization scheme. Variational parameters are updated via NGD, while model hyperparameters (such as kernel lengthscales) are updated using a standard optimizer like Adam. Additionally, they identify a robust step-size schedule: starting with a small step and increasing it toward 1.0—the theoretical optimum for conjugate cases—leads to remarkably stable and rapid convergence.
Results: Faster Convergence and Enhanced Stability
The proposed method was benchmarked across a variety of regression and classification tasks, including large-scale datasets with over 500,000 samples. The results demonstrate that NGD consistently outperforms standard methods in wall-clock time, often reaching the optimal Evidence Lower Bound (ELBO) in a fraction of the time required by Adam or GD.
Crucially, NGD provides essential stability. In "stiff" problems like the Naval dataset, standard gradients fail to converge regardless of the learning rate, whereas NGD succeeds. Furthermore, the method is invariant to parameterization; whether using mean/variance or natural parameters, NGD follows the same optimization path, simplifying model development.
Conclusion
This work demonstrates that Natural Gradient Descent is a highly practical and often necessary tool for non-conjugate Gaussian Process models. By accounting for the geometry of probability distributions, NGD offers faster convergence, greater robustness to parameterization, and improved stability in challenging optimization landscapes. Future research into automated step-size selection could further enhance the accessibility of these methods.
Summary updated