Sparse Gaussian Processes with Inducing Points

This direction is merged into another direction: inducing point approximation. Its last version stays readable and citable here.

Version 1 of 1234 papers261 conceptsfrozen 2026-10-04

Gaussian Processes (GPs) are the "gold standard" for Bayesian non-parametric modeling because they provide elegant, closed-form uncertainty estimates. However, they suffer from a notorious "cubic bottleneck": calculating the relationship between $N$ data points requires inverting an $N \times N$ covariance matrix, a process that scales at $O(N^3)$. For a dataset of a million points, this is computationally impossible.

The research direction of Sparse Gaussian Processes solves this by replacing the full dataset with a small set of "inducing points" (or pseudo-inputs). These points act as a compressed summary of the entire function space, reducing complexity from cubic to linear relative to the total number of observations.

The Intuition of Inducing Points as Skeletons

Imagine trying to describe the shape of a mountain range. You could provide the GPS coordinates of every single grain of sand (the full GP approach), but that is redundant and overwhelming. Instead, you could provide the coordinates of just the peaks and valleys—the "landmarks" that define the overall structure.

In Sparse GPs, these landmarks are the inducing points. As Yves-Laurent Kom Samo demonstrates in Scalable Nonparametric Bayesian Inference on Point Processes with Gaussian Processes, by optimizing a small set of these points, we can represent the underlying function with high fidelity while shifting the computational cost from the total number of observations to a fixed, much smaller number of reference points.

Variational Inference and Scalability

The breakthrough that allowed these models to scale to millions of points was the marriage of sparse approximations with variational inference. Instead of trying to find the "perfect" GP, researchers define a simpler, approximate distribution and minimize the "distance" (KL divergence) between it and the true posterior.

Chris Lloyd, Tom Gunter, and Michael A. Osborne introduced Variational Inference for Gaussian Process Modulated Poisson Processes, which was the first fully variational scheme for point processes. Their work showed that by using inducing points to summarize the latent process, inference could achieve $O(N)$ linear scaling. This allowed models to handle large-scale event data without the precision loss associated with "binning" data into a grid.

Similarly, José Miguel Hernández-Lobato showed in Scalable Gaussian Process Classification via Expectation Propagation that tying the latent function to inducing points allows for stochastic minibatch training, much like how modern neural networks are trained.

Deep and Structured Extensions

Once the scaling problem was addressed, researchers began stacking these models to create Deep Gaussian Processes (DGPs). In their foundational paper, Deep Gaussian Processes, Andreas C. Damianou and Neil D. Lawrence used auxiliary variables (inducing points) to make multi-layered GP architectures computationally feasible. This allowed GPs to learn hierarchical features similar to deep neural networks while maintaining rigorous probabilistic foundations.

This concept has been extended into specialized domains:

Why It Works: The Information Summary

The reason inducing points work so effectively is that they don't just subsample the data; they are learned parameters that summarize the total information. As Aidan Scannell and Paul Edmund Chang explain in Sparse Function-space Representation of Neural Networks, these points carry information derived from the full set of gradients and curvatures of the training data.

By projecting the complexity of a massive dataset onto a limited set of inducing points, researchers have transformed Gaussian Processes from a "small data" curiosity into a scalable tool for modern machine learning, capable of providing uncertainty-aware predictions on datasets with millions of observations.

Papers in this version

Showing 60 of 234 papers.

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_sparse_gaussian_processes_with_inducing__1d714306,
  title = {Sparse Gaussian Processes with Inducing Points},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sparse-gaussian-processes-with-inducing-points/rdr_1d7143066e2c4d1e9c8822cc06900ead},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_sparse_gaussian_processes_with_inducing__1d714306_v1,
  title = {Sparse Gaussian Processes with Inducing Points},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/sparse-gaussian-processes-with-inducing-points/rdr_1d7143066e2c4d1e9c8822cc06900ead/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}