Gaussian Processes (GPs) are widely considered the gold standard for Bayesian non-parametric modeling due to their ability to provide principled uncertainty estimates. However, they famously suffer from a cubic scaling bottleneck—$O(N^3)$—making them impractical for modern, large-scale datasets.
In their paper, researchers from Tsinghua University and RIKEN introduce GPNet, a scalable algorithm for training inference networks to approximate Gaussian Process posteriors. By treating the training process as an adaptive Bayesian filter in function space, GPNet overcomes the limitations of traditional sparse GPs and enables the use of deep architectures, such as Convolutional Neural Networks (CNNs), for GP inference on massive datasets.
Background: The Scalability Bottleneck
The standard solution to GP scalability is the Sparse GP, which uses a small set of "inducing points" to summarize the dataset. While effective, sparse GPs have two major flaws:
- Limited Flexibility: The complexity of the posterior is strictly capped by the number of inducing points.
- Optimization Hurdles: Finding the optimal locations for these points is a difficult, non-convex optimization problem.
An alternative approach is to use inference networks—neural networks designed to output function values that mimic the GP posterior. However, training these networks with minibatches is notoriously difficult. Because a minibatch only exposes the model to a tiny slice of the data at a time, the network struggles to maintain meaningful correlations across the entire function space. This often leads to poor uncertainty estimates or overfitting to local noise.
Methodology: The GPNet Approach
To solve these training challenges, the authors propose GPNet, which utilizes a Stochastic Functional Mirror Descent algorithm.
The "Adaptive Filter" Approach
Instead of performing standard gradient descent on network weights, GPNet performs updates in the function space. Each iteration follows a two-step logic:
- The Mirror Step: The algorithm calculates a "target" distribution (acting as a teacher) by taking the current model and updating it with a single minibatch using a closed-form Gaussian update.
- The Tracking Step: The inference network (the student) is then trained to "track" or move toward this target distribution by minimizing the Kullback-Leibler (KL) divergence at a set of random measurement points.
Flexible Architectures
GPNet is agnostic to the specific network architecture. The authors demonstrate its versatility using:
- Random Feature Expansion (RFE): Shallow networks that approximate kernels via Monte Carlo sampling of spectral features.
- Neural Tangent Kernels (NTK): A formulation that treats deep, infinite-width neural networks as GPs, allowing the use of architectures like CNNs for GP inference.
Key Insights
- Decoupling Capacity from Computation: In sparse GPs, the number of inducing points ($M$) limits both the computational speed and the model's expressiveness. In GPNet, the neural network provides the expressiveness, while the number of measurement points ($M$) only affects the variance of the training signal.
- Global vs. Local Objectives: By using the mirror descent framework, GPNet maintains a shared global objective. This prevents the model from "forgetting" the rest of the function when it processes a new minibatch, a common failure mode in previous functional inference attempts.
Experimental Results
The authors evaluated GPNet across a diverse range of benchmarks, from synthetic data to large-scale regression and image classification.
- Better Fit with Fewer Points: On the synthetic Snelson dataset, GPNet recovered the exact GP prediction with as few as 2 measurement points. In contrast, standard Sparse Variational GPs (SVGP) required significantly more inducing points to achieve similar results.
- Superior Scaling: On the Airline Delay dataset, containing 5.9 million flight records, GPNet achieved a lower Root Mean Square Error (RMSE) than both SVGP and other state-of-the-art functional inference methods.
- State-of-the-Art CNN-GPs: On the CIFAR-10 image dataset, GPNet achieved a 24.63% test error using a CNN-GP prior. This outperformed previous approximate inference methods and demonstrated that GPNet can successfully handle high-dimensional input spaces where inducing-point methods often fail.
Limitations and Future Work
While highly scalable, GPNet is not without limitations:
- Computational Overhead: The algorithm still requires an $O(M^3)$ matrix inversion per iteration (where $M$ is the number of measurement points). While $M$ can be kept small, this remains a computational cost to consider.
- Hyperparameter Tuning: The mirror descent approach introduces new hyperparameters, such as the learning rate schedule, which may require careful tuning for different tasks.
- Jacobian Costs: For the deep NTK variants, computing gradients requires backpropagating through Jacobians, which is more computationally expensive than standard neural network training. Exploring more efficient gradient estimators is a promising direction for future research.
Summary updated