Abstract
Inducing Gaussian Process Networks (IGN) address the scalability and complexity challenges of Gaussian Processes (GPs) by situating inducing points within a learned feature space rather than the raw input space. This approach enables GPs to handle high-dimensional and structured data—such as images, graphs, and text—while preserving robust uncertainty estimation. We demonstrate that IGNs achieve state-of-the-art performance on various regression and classification benchmarks, offering a domain-agnostic framework for scalable Bayesian deep learning.
Introduction
Gaussian Processes (GPs) are widely regarded as the "gold standard" for uncertainty estimation in machine learning due to their principled Bayesian framework. However, they face two significant hurdles in modern applications:
- Scalability: The $O(n^3)$ computational complexity of standard GPs makes them impractical for large-scale datasets.
- Complex Input Domains: Traditional "Sparse GPs" utilize a small set of "inducing points" to approximate the full GP. However, these points typically reside in the input space. For high-dimensional or structured data like images, text, or graphs, defining and optimizing representative input points is mathematically challenging and often lacks intuitive meaning.
Methodology: Inducing Points in Latent Space
The core innovation of Inducing Gaussian Process Networks (IGN) is the relocation of inducing points from the raw input space $\mathcal{X}$ to a latent feature space $\mathbb{R}^d$.
The framework consists of several key components:
- Feature Embedding: A neural network $g$ maps complex inputs (like images or graphs) into a lower-dimensional feature space.
- Inducing Points ($Z$): Instead of optimizing representative inputs, the model optimizes a set of vectors $Z$ directly in the feature space.
- Pseudo-Labels: A separate function $r$ assigns targets to these inducing points, facilitating the GP approximation.
- End-to-End Training: IGNs are trained using mini-batch gradient descent to jointly optimize the neural network weights, inducing points, and kernel parameters. A key theoretical result shows that the gradient for the inducing points is decoupled from the gradient of the embedding function, enhancing optimization stability.
Experimental Evaluation
The authors evaluated IGN across diverse tasks, ranging from standard regression to complex structured data classification.
Regression and Computer Vision
On regression benchmarks like the Levy and Borehole datasets, IGN significantly outperformed state-of-the-art methods such as Sparse Gaussian Process Regression (SGPR) and Stochastic Variational Gaussian Processes (SVGP) in terms of Root Mean Square Error (RMSE).
In computer vision, IGN demonstrated strong performance on MNIST (98.0% accuracy) and CIFAR-10 (51.03% accuracy). These results surpass other deep kernel competitors, including Deep Gaussian Processes (DGP) and Neural Network Gaussian Processes (NNGP).
Structured Data
A major advantage of IGN is its domain agnosticism. By mapping inputs to a vector space, the GP can process various data types seamlessly. The model was tested on:
- IMDB-TEXT: Sentiment analysis on movie reviews.
- IMDB-GRAPH: Genre classification based on social collaboration networks.
In both cases, IGN outperformed standard graph neural networks and text classifiers reported in previous literature.
Discussion and Key Insights
IGN preserves the classic GP property where model variance increases for data points far from the learned inducing points in feature space, providing reliable uncertainty estimates. Furthermore, while inducing points live in the feature space, they can be "mapped back" to the input space via gradient descent to visualize what the model considers a "representative" input (e.g., generating prototypical digits).
Limitations and Future Work
Current limitations include the use of a "one-vs-all" approach for multi-class classification and a reliance on maximum likelihood estimation. Future work aims to develop a fully variational setting to improve probabilistic rigor and conduct deeper investigations into the calibration of uncertainty estimates.
Summary updated