Gaussian Processes (GPs) are often described as the "gold standard" for probabilistic regression. Unlike standard neural networks, which provide a single point estimate, GPs define a distribution over functions, offering elegant, non-parametric uncertainty estimates. However, they carry a notorious "cubic bottleneck": calculating the posterior requires inverting a covariance matrix, an operation that scales at $O(N^3)$ with the number of data points $N$. For a dataset of 100,000 points, this is computationally prohibitive; for millions, it is impossible.
To bring the benefits of GPs to the era of big data, researchers have developed a framework that mirrors the scalability of deep learning: Stochastic Variational Gaussian Processes (SVGP). By combining sparse approximations with variational inference, this direction allows GPs to scale to millions of observations while maintaining their principled Bayesian foundations.
The Inducing Point Approximation
The first step in breaking the cubic bottleneck is to stop trying to account for every data point simultaneously. Instead, we summarize the entire dataset using a small set of $M$ "inducing points" (where $M \ll N$).
As explained by Haitao Liu and Yew Soon Ong in Understanding and Comparing Scalable Gaussian Process Regression for Big Data, these inducing points act as a "sketch" of the data. Rather than computing the relationships between all $N$ points, we compute the relationships between the $N$ points and the $M$ inducing points. This reduces the core computational complexity from $O(N^3)$ to $O(NM^2)$, making the model linear with respect to the total dataset size.
Mini-batching via Variational Inference
While inducing points make the math faster, they don't inherently allow for the "mini-batch" training that makes deep learning so successful. Standard GPs require the entire dataset to be present to calculate the marginal likelihood.
The breakthrough comes from Variational Inference. By treating the values at the inducing points as latent variables, researchers like Yarin Gal and Mark van der Wilk demonstrated in Variational Inference in Sparse Gaussian Process Regression and Latent Variable Models - a Gentle Tutorial that the Evidence Lower Bound (ELBO)—the objective function we maximize—can be factorized.
Because the ELBO becomes a sum of terms over individual data points, we can use Stochastic Gradient Descent (SGD). We sample a small mini-batch of data, calculate the gradient of the ELBO for that batch, and update the inducing points and kernel hyperparameters. This "stochastic" approach decouples the training cost from the total number of observations, allowing models to process millions of points, as seen in the work of Florian Wenzel on Efficient Gaussian Process Classification Using Pólya-Gamma Data Augmentation.
Deep and Heterogeneous Architectures
Once GPs are scalable, they can be integrated into more complex architectures. This has led to several specialized directions:
- Deep Gaussian Processes (DGPs): By stacking GPs in layers, researchers like Melih Kandemir in Asymmetric Transfer Learning with Deep Gaussian Processes create hierarchical models that capture complex, non-stationary patterns while using "doubly stochastic" inference to remain efficient.
- GP-VAEs: Frameworks like the one proposed by Matthew Ashman in Scalable Gaussian Process Variational Autoencoders use GPs to model the latent space of a Variational Autoencoder, allowing the model to capture temporal or spatial correlations in high-dimensional data like video or sensor streams.
- Multi-output and Heterogeneous Models: Pablo Moreno-Muñoz has shown in Heterogeneous Multi-output Gaussian Process Prediction that these scalable frameworks can jointly model different types of data (e.g., continuous, binary, and counts) by sharing a latent GP structure.
Why It Works: The Intuition of Local Influence
The success of these scalable methods relies on the intuition that in a large dataset, not every point is equally informative for every prediction. Geoff Pleiss and Luhuan Wu argue in Variational nearest neighbor Gaussian process that for spatial data, local dependencies are often more important than global ones.
By using inducing points or nearest-neighbor structures, we are essentially building a "global summary" that handles the broad trends, while the variational framework allows the model to ignore the "noise" of millions of redundant points. This makes it possible to apply GPs to massive real-world problems, from predicting influenza spread across the US to disentangling biological signals in single-cell RNA sequencing data involving hundreds of thousands of cells.