Scalable Gaussian Processes via Stochastic Variational Inference

This direction is merged into another direction: inducing point approximation. Its last version stays readable and citable here.

Version 1 of 185 papers91 conceptsfrozen 2026-10-04

Gaussian Processes (GPs) are often described as the "gold standard" for probabilistic regression. Unlike standard neural networks, which provide a single point estimate, GPs define a distribution over functions, offering elegant, non-parametric uncertainty estimates. However, they carry a notorious "cubic bottleneck": calculating the posterior requires inverting a covariance matrix, an operation that scales at $O(N^3)$ with the number of data points $N$. For a dataset of 100,000 points, this is computationally prohibitive; for millions, it is impossible.

To bring the benefits of GPs to the era of big data, researchers have developed a framework that mirrors the scalability of deep learning: Stochastic Variational Gaussian Processes (SVGP). By combining sparse approximations with variational inference, this direction allows GPs to scale to millions of observations while maintaining their principled Bayesian foundations.

The Inducing Point Approximation

The first step in breaking the cubic bottleneck is to stop trying to account for every data point simultaneously. Instead, we summarize the entire dataset using a small set of $M$ "inducing points" (where $M \ll N$).

As explained by Haitao Liu and Yew Soon Ong in Understanding and Comparing Scalable Gaussian Process Regression for Big Data, these inducing points act as a "sketch" of the data. Rather than computing the relationships between all $N$ points, we compute the relationships between the $N$ points and the $M$ inducing points. This reduces the core computational complexity from $O(N^3)$ to $O(NM^2)$, making the model linear with respect to the total dataset size.

Mini-batching via Variational Inference

While inducing points make the math faster, they don't inherently allow for the "mini-batch" training that makes deep learning so successful. Standard GPs require the entire dataset to be present to calculate the marginal likelihood.

The breakthrough comes from Variational Inference. By treating the values at the inducing points as latent variables, researchers like Yarin Gal and Mark van der Wilk demonstrated in Variational Inference in Sparse Gaussian Process Regression and Latent Variable Models - a Gentle Tutorial that the Evidence Lower Bound (ELBO)—the objective function we maximize—can be factorized.

Because the ELBO becomes a sum of terms over individual data points, we can use Stochastic Gradient Descent (SGD). We sample a small mini-batch of data, calculate the gradient of the ELBO for that batch, and update the inducing points and kernel hyperparameters. This "stochastic" approach decouples the training cost from the total number of observations, allowing models to process millions of points, as seen in the work of Florian Wenzel on Efficient Gaussian Process Classification Using Pólya-Gamma Data Augmentation.

Deep and Heterogeneous Architectures

Once GPs are scalable, they can be integrated into more complex architectures. This has led to several specialized directions:

Why It Works: The Intuition of Local Influence

The success of these scalable methods relies on the intuition that in a large dataset, not every point is equally informative for every prediction. Geoff Pleiss and Luhuan Wu argue in Variational nearest neighbor Gaussian process that for spatial data, local dependencies are often more important than global ones.

By using inducing points or nearest-neighbor structures, we are essentially building a "global summary" that handles the broad trends, while the variational framework allows the model to ignore the "noise" of millions of redundant points. This makes it possible to apply GPs to massive real-world problems, from predicting influenza spread across the US to disentangling biological signals in single-cell RNA sequencing data involving hundreds of thousands of cells.

Papers in this version

Showing 60 of 85 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_scalable_gaussian_processes_via_stochast_2ced4d7f,
  title = {Scalable Gaussian Processes via Stochastic Variational Inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scalable-gaussian-processes-via-stochastic-variational-inference/rdr_2ced4d7fbff64ae1b0faec7a95aeb547},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_scalable_gaussian_processes_via_stochast_2ced4d7f_v1,
  title = {Scalable Gaussian Processes via Stochastic Variational Inference},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/scalable-gaussian-processes-via-stochastic-variational-inference/rdr_2ced4d7fbff64ae1b0faec7a95aeb547/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}