A Tutorial on Sparse Gaussian Processes and Variational Inference

Felix Leibfried, Vincent Dutordoir, S. T. John, Nicolas Durrande

Introduction

Gaussian Processes (GPs) represent a cornerstone of Bayesian non-parametrics, offering a "gold standard" for regression tasks due to their ability to provide principled uncertainty estimates. Unlike standard neural networks, which often provide point estimates, GPs define a distribution over functions, allowing for a mathematically rigorous measure of predictive confidence. However, the classical GP framework suffers from a significant computational limitation: its complexity scales cubically ($O(N^3)$) with the number of data points $N$. This scaling makes exact GPs intractable for modern large-scale datasets.

This paper by Leibfried et al. serves as a comprehensive tutorial on overcoming these limitations. By integrating Sparse Gaussian Processes with Variational Inference (VI), the authors provide a roadmap for scaling GPs to large, complex, and non-Gaussian datasets, effectively bridging the gap between classical Bayesian statistics and modern deep learning.

The Scalability and Flexibility Bottleneck

The primary challenge in exact GP regression is the requirement to invert an $N \times N$ covariance matrix to calculate the posterior distribution. For $N=10,000$, this is manageable; for $N=1,000,000$, the computational and memory requirements become prohibitive.

Beyond scalability, exact GPs are traditionally limited to Gaussian likelihoods. In scenarios involving classification (Bernoulli likelihood) or heavy-tailed noise, the posterior distribution lacks a closed-form solution. To address these issues, the tutorial explores methods that transform the inference problem into an optimization task, enabling the use of stochastic gradient descent and flexible likelihood functions.

Methodological Framework

Sparse GPs and Inducing Points

To mitigate the $O(N^3)$ cost, the authors employ the concept of "inducing points." Instead of conditioning the model on all $N$ observations, a smaller set of $M$ pseudo-training examples (where $M \ll N$) is used to represent the function. This reduction effectively lowers the computational complexity to $O(NM^2)$, making the model scalable to much larger datasets.

Variational Inference (VI)

The authors utilize Variational Inference to treat the selection of inducing point locations and hyperparameters as an optimization problem. By defining and maximizing the Evidence Lower Bound (ELBO), the model can simultaneously learn the optimal "summary" of the data and the underlying kernel parameters. This approach allows for the application of GPs to non-Gaussian likelihoods by approximating the true posterior with a simpler, tractable distribution.

Deep and Latent Variable Gaussian Processes

A significant contribution of the tutorial is the discussion of Deep Gaussian Processes (DGPs). By stacking GP layers, the model can learn hierarchical representations, where the output of one layer serves as the input to the next. This architecture allows the model to capture non-smooth, abrupt changes in data that a single "shallow" GP would typically over-smooth. Furthermore, the inclusion of latent variables (LV-DGPs) enables the modeling of multimodal data, where a single input might correspond to multiple possible output values.

Empirical Evaluation: The "DGP" Dataset

The tutorial evaluates these methods using a synthetic multimodal dataset where data points are arranged to form the letters "DGP." This benchmark tests the models' ability to recover complex, overlapping, and non-smooth shapes.

The results highlight the limitations of traditional approaches:

  • Shallow Sparse GPs: These models failed to capture the sharp edges of the letters, resulting in a blurred "average" of the data points.
  • Deep GPs (DGP): The hierarchical structure allowed for a significantly better fit, capturing the distinct shapes of the letters more accurately.
  • Latent-Variable Deep GPs (LV-DGP): This model provided the highest fidelity, successfully resolving the multimodality inherent in the overlapping parts of the letters (e.g., the vertical and curved sections of the "D").

Key Insights for Practitioners

The authors identify several critical technical nuances for implementing these models:

  • The Double Dependence: Unlike variational inference in standard neural networks, the approximate posterior in sparse GPs depends on both the variational parameters (inducing points) and the generative hyperparameters (kernel parameters).
  • Numerical Stability: The use of "whitening" transformations—specifically Cholesky decomposition—is emphasized as a vital technique for stabilizing the optimization of sparse GPs.
  • Importance Weighting: While computationally more expensive, importance sampling can provide a tighter ELBO, leading to more reliable model training and better marginal likelihood estimates.

Conclusion

Leibfried et al. demonstrate that Gaussian Processes, when augmented with sparse approximations and variational techniques, remain a powerful and scalable tool for modern machine learning. While Deep GPs introduce additional complexity—requiring Monte Carlo sampling to propagate uncertainty—they offer the flexibility needed to model the non-smooth and multimodal phenomena found in real-world data. Future research continues to explore more sophisticated interdomain features and the optimal integration of latent variables across hierarchical layers.

Summary updated

Abstract

Gaussian processes (GPs) provide a framework for Bayesian inference that can offer principled uncertainty estimates for a large range of problems. For example, if we consider regression problems with Gaussian likelihoods, a GP model enjoys a posterior in closed form. However, identifying the posterior GP scales cubically with the number of training examples and requires to store all examples in memory. In order to overcome these obstacles, sparse GPs have been proposed that approximate the true posterior GP with pseudo-training examples. Importantly, the number of pseudo-training examples is user-defined and enables control over computational and memory complexity. In the general case, sparse GPs do not enjoy closed-form solutions and one has to resort to approximate inference. In this context, a convenient choice for approximate inference is variational inference (VI), where the problem of Bayesian inference is cast as an optimization problem -- namely, to maximize a lower bound of the log marginal likelihood. This paves the way for a powerful and versatile framework, where pseudo-training examples are treated as optimization arguments of the approximate posterior that are jointly identified together with hyperparameters of the generative model (i.e. prior and likelihood). The framework can naturally handle a wide scope of supervised learning problems, ranging from regression with heteroscedastic and non-Gaussian likelihoods to classification problems with discrete labels, but also problems with multidimensional labels. The purpose of this tutorial is to provide access to the basic matter for readers without prior knowledge in both GPs and VI. A proper exposition to the subject enables also access to more recent advances (like importance-weighted VI as well as interdomain, multioutput and deep GPs) that can serve as an inspiration for new research ideas.
BibTeX
@misc{leibfried2020tutorial,
  author = {Felix Leibfried and Vincent Dutordoir and S. T. John and Nicolas Durrande},
  title = {{A Tutorial on Sparse Gaussian Processes and Variational Inference}},
  year = {2020},
  eprint = {2012.13962},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.13962},
}
Figure