Scalable Training of Inference Networks for Gaussian-Process Models

Jiaxin Shi, Mohammad Emtiyaz Khan, Jun Zhu

Abstract

Gaussian Processes (GPs) are valued for their principled uncertainty quantification, yet their $O(N^3)$ computational complexity limits their application to large-scale datasets. While sparse GP methods using "inducing points" offer a solution, they introduce a capacity bottleneck where model flexibility is tied to computational budget. This paper presents GPNet, a framework that leverages stochastic inference networks and functional mirror descent to achieve scalable GP inference without compromising model flexibility.

Introduction

The standard approach to scaling GPs involves approximating the full dataset with a small set of $M$ inducing points. However, the model's expressive power is fundamentally constrained by $M$. Increasing $M$ to capture complex patterns leads to prohibitive computational costs. GPNet addresses this by using an inference network to represent the function distribution, decoupling the number of measurement points used during training from the model's overall capacity.

The GPNet Framework

GPNet treats the training of an inference network as a tracking problem in function space, utilizing a stochastic functional mirror-descent algorithm to overcome the limitations of weight-space optimization.

Schematic of the GPNet framework approximating a Gaussian Process posterior with an inference network.

Stochastic Functional Mirror Descent

Instead of optimizing network weights directly via standard gradient descent, GPNet performs updates in the space of functions. By using a KL-divergence-based step, the algorithm is theoretically guaranteed to converge toward the true GP posterior. To handle the intractability of the functional gradient, the authors employ a "bootstrap trick," where the network's current state serves as a prior for subsequent updates—an approach similar in spirit to temporal difference learning in Reinforcement Learning.

Measurement Points vs. Inducing Points

A critical innovation in GPNet is the use of "measurement points." Unlike inducing points in sparse GPs, these points are sampled stochastically from the input space at each iteration to compute gradients. They do not define the model's capacity; rather, they control the variance of the training signal. This allows the underlying neural network to maintain high flexibility even when using very few measurement points, effectively fixing the "minibatch issue" where previous methods failed to preserve global correlations.

Experimental Results

The authors evaluated GPNet across synthetic benchmarks, large-scale regression tasks, and image classification, demonstrating its robustness and scalability.

Comparison of GPNet and Sparse GP (SVGP) on the Snelson dataset.

Synthetic and Large-Scale Regression

On the Snelson dataset, GPNet successfully recovered the exact GP posterior with as few as $M=2$ measurement points, a task where traditional Stochastic Variational GPs (SVGP) failed significantly. In large-scale tests on the Airline Delay dataset (700,000 training samples), GPNet achieved superior RMSE compared to established functional baselines and sparse GP methods.

Image Classification with CNN-GPs

GPNet enables the scalable training of GPs with convolutional kernels induced by infinite-width Bayesian ConvNets. On the CIFAR-10 dataset, GPNet achieved a test error of 24.63%, significantly outperforming previous approximate CNN-GP methods which typically report errors in the 35-44% range.

Conclusion

GPNet provides a robust path forward for scaling Gaussian Processes by moving inference into the function space via neural networks. By decoupling computational constraints from model capacity, it allows for the deployment of sophisticated GP priors on massive datasets, bridging the gap between deep learning flexibility and Bayesian principled uncertainty.

Summary updated

BibTeX
@inproceedings{shi2019scalable,
  author = {Jiaxin Shi and Mohammad Emtiyaz Khan and Jun Zhu},
  title = {{Scalable Training of Inference Networks for Gaussian-Process Models}},
  booktitle = {International Conference on Machine Learning},
  year = {2019},
  url = {https://openreview.net/forum?id=H1ZO3jZdZB},
}
Figure