The Rényi Gaussian Process

Xubo Yue, RkarXiv 2019 · arXiv · OpenReview

Abstract

We introduce an alternative closed form lower bound on the Gaussian process ($\mathcal{GP}$) likelihood based on the Rényi $\alpha$-divergence. This new lower bound, the $\alpha$-ELBO, can be viewed as a convex combination of the Nyström approximation and the exact $\mathcal{GP}$. The key advantage of this bound is its capability to control and tune the enforced regularization on the model, thus serving as a generalization of traditional variational $\mathcal{GP}$ regression. From a theoretical perspective, we provide the convergence rate and risk bound for inference using our proposed approach. Experiments on real data show that the proposed algorithm delivers significant improvements over several established $\mathcal{GP}$ inference methods.

Introduction

Gaussian Processes (GPs) are powerful tools for uncertainty quantification, yet they present a significant challenge in inference. Exact inference is computationally expensive ($O(N^3)$) and susceptible to overfitting in the presence of noise. Conversely, Variational Inference (VI) optimizes the Evidence Lower Bound (ELBO) to provide scalability and regularization, but a tighter ELBO is not always optimal; it can sometimes reduce the signal-to-noise ratio. Currently, researchers must choose between these two extremes. The Rényi Gaussian Process bridges this gap by providing a mathematical "knob"—the $\alpha$ parameter—to precisely dial in the required amount of regularization for a given dataset.

Methodology

The proposed framework introduces the $\alpha$-ELBO ($L_\alpha$), derived from the Rényi $\alpha$-divergence. This bound acts as a convex combination of the exact GP covariance and the Nyström low-rank approximation used in sparse GPs.

By adjusting $\alpha$, the model interpolates between different regimes:

  • $\alpha \to 1$: Recovers the standard KL-based Variational GP (SGP).
  • $\alpha \to 0$: Recovers the exact GP likelihood (EGP).

To ensure scalability to large datasets (over 100,000 samples), the authors utilize the Blackbox Matrix-Matrix multiplication (BBMM) algorithm. This allows the model to leverage conjugate gradients and GPU acceleration, maintaining the computational complexity of state-of-the-art exact GP implementations while benefiting from the tunable variational framework.

Experimental Results

The Rényi GP was evaluated against Exact GP (EGP), Sparse Variational GP (SGP), and Power Expectation Propagation (PEP) using both simulated functions and high-dimensional real-world datasets.

Simulated Benchmarks

On synthetic tasks like the Gramacy & Lee and Griewank-4 functions, the Rényi GP demonstrates how varying $\alpha$ allows for performance optimization. As seen in Figure 1, intermediate values of $\alpha$ often outperform both the exact and standard variational baselines.

Comparative performance on simulated benchmarks (Gramacy & Lee and Griewank-4) showing the effect of varying $\alpha$ against baseline methods.

Real-World Performance

In real-world regression benchmarks, the Rényi GP consistently achieved the lowest RMSE. The performance gains were particularly notable on noisy datasets. For instance, on the "Battery" dataset, the Rényi GP reached an RMSE of 9.90, compared to 20.16 for EGP and 29.96 for SGP.

RMSE comparison on real-world datasets showing the Rényi GP's consistent outperformance and the stability of the optimal $\alpha$.

Key Insights and Conclusion

Empirical results suggest a "sweet spot" for the $\alpha$ parameter. Across diverse datasets, the optimal $\alpha$ consistently fell between 0.45 and 0.55. This indicates that a moderate degree of variational "blurring" serves as a beneficial regularizer, preventing the model from overfitting to noise while retaining more predictive signal than standard ELBO-based methods.

The Rényi GP provides a unified framework that encapsulates existing GP inference methods as special cases. While currently focused on Gaussian likelihoods, future extensions to non-Gaussian cases and automated $\alpha$ selection represent promising directions for further research.

Summary updated

Abstract

We introduce an alternative closed form lower bound on the Gaussian process ($\mathcal{GP}$) likelihood based on the R\'enyi $\alpha$-divergence. This new lower bound can be viewed as a convex combination of the Nystr\"om approximation and the exact $\mathcal{GP}$. The key advantage of this bound, is its capability to control and tune the enforced regularization on the model and thus is a generalization of the traditional variational $\mathcal{GP}$ regression. From a theoretical perspective, we provide the convergence rate and risk bound for inference using our proposed approach. Experiments on real data show that the proposed algorithm may be able to deliver improvement over several $\mathcal{GP}$ inference methods.
BibTeX
@misc{yue2019renyi,
  author = {Xubo Yue and RK},
  title = {{The Rényi Gaussian Process}},
  year = {2019},
  eprint = {1910.06990},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1910.06990},
}
Figure