TL;DR
Inducing Gaussian Process Networks (IGN) introduce a scalable, end-to-end framework for Gaussian processes by placing inducing points directly into an unconstrained neural feature space rather than the input space. Combined with an amortized pseudo-label function and mini-batch maximum likelihood estimation, IGN sidesteps the exponential memory footprint of grid-based deep kernels while seamlessly handling complex data types like images, text, and graphs.
What Problem Does This Paper Address?
Standard Gaussian processes (GPs) provide elegant Bayesian uncertainty estimation, but their computational complexity scales cubically, $\mathcal{O}(n^3)$, with sample size. Sparse GP approximations mitigate this bottleneck using a smaller set of $m$ inducing points as proxies for the data. However, traditional sparse GPs define these inducing points directly in the raw input space $\mathcal{X}$. This causes major issues when inputs are high-dimensional, discrete, or irregularly structured (such as graphs, raw text, or point clouds).
While deep kernel learning (DKL) combines neural feature extractors with GPs, standard formulations often place inducing points on rigid multi-dimensional lattices that scale exponentially with feature dimensionality, require pre-training, or rely on complex doubly-stochastic variational approximations. IGN addresses this challenge by providing an unconstrained, end-to-end trainable inducing-point framework in the latent feature space.
Methods
IGN integrates deep neural embeddings with sparse GP regression and classification through several key components:
- Unconstrained Feature-Space Inducing Points: A neural network $g_{\theta_g}: \mathcal{X} \to \mathbb{R}^d$ maps inputs into a $d$-dimensional latent space. A set of $m$ inducing points $Z = \{z_1, \dots, z_m\}$ lives directly in $\mathbb{R}^d$, completely independent of the structure of $\mathcal{X}$.
- Amortized Pseudo-Label Function: Rather than assigning independent variational parameters to each inducing point, IGN defines a shared, linear pseudo-label mapping $r_{\theta_r}: \mathbb{R}^d \to \mathbb{R}$. This provides an amortized proxy for inducing observation values.
- Gradient Separation (Proposition 1): The authors prove that the gradients of the marginal log-likelihood with respect to the inducing points $Z$ and the pseudo-label weights $\theta_r$ do not depend on $\nabla_{\theta_g} g$. This decouples the direct updates of inducing points from potentially noisy or ill-conditioned neural network gradients.
- End-to-End Mini-Batch Optimization: The parameters $\theta = \{Z, \theta_g, \theta_r, \sigma_\epsilon\}$ are optimized jointly via mini-batch gradient descent on the GP marginal log-likelihood (using a Laplace approximation for classification tasks), requiring no stage-wise pre-training.
Input x ──[ Neural Net g(·) ]──> Latent Vector g(x) ──┐
├──> Base Kernel k(·,·) ──> GP Posterior
Inducing Points Z ─────────────> Latent Vectors Z ──┤
└────[ Linear r(·) ]───────> Pseudo-labels r ─────┘
Datasets
The authors evaluate IGN across a variety of modalities:
- Simulated Regression: Levy, Griewank, and Borehole (from the Virtual Library of Simulation Experiments).
- Real-World Tabular Regression: Protein, PM2.5, Energy, Bike-Hour, and Query (from the UCI Machine Learning Repository; sizes up to 1,000,000 samples).
- Vision: Toy-MNIST (classes 5 vs. 6 for qualitative analysis), full MNIST, and CIFAR-10.
- Structured Domains:
- IMDB-TEXT: 50,000 sentiment analysis movie reviews paired with a Transformer feature extractor.
- IMDB-GRAPH: 1,000 movie collaboration ego-graphs categorized by genre (Action vs. Romance) paired with a graph neighborhood extractor.
Key Insights and Ideas
- Decoupling Inducing Points from Input Topologies: Because inducing points exist only in $\mathbb{R}^d$, the exact same sparse GP formulation works uniformly across tabular data, images, text, and graphs without needing domain-specific kernel designs.
- Breaking the DKL Memory Wall: Traditional DKL relies on structured grid layouts that explode in memory beyond very few dimensions or points (failing past 8 inducing points in benchmark runs). IGN scales easily to hundreds of inducing points (e.g., 512) while keeping per-epoch compute times flat.
- Latent Space Explainability: Even though inducing points are abstract feature vectors, they can be interpreted by mapping them back to the nearest training instances in feature space (or gradient-optimizing synthetic pseudo-images). In text and digit tasks, these pseudo-prototypes clearly separate along class boundaries.
- Predictive Variance Monotonicity: The authors prove and empirically confirm that test variance consistently decreases as the number of feature-space inducing points increases, ensuring reliable uncertainty shrinkage.
Key Results
- Regression Benchmarks: IGN consistently matches or outperforms baselines (sgGP, DKL, DIWP, EGP, SGPR, SVGP). Notably, IGN achieves 0.00 RMSE on Borehole (vs. 0.17 for sgGP) and 0.01 RMSE on Bike-Hour (vs. 0.22 for sgGP).
- Classification: Evaluated via a one-vs-all scheme, IGN achieves higher test accuracy than Deep GPs (DGP), neural network GPs (NNGP), and stochastic variational DKL (SVI-DKL):
- MNIST: 98.0% accuracy (compared to 96.5% for DGP and 97.4% for SVI-DKL).
- CIFAR-10: 51.0% accuracy (compared to 46.8% for DGP and 48.1% for SVI-DKL).
- Structured Text & Graphs: IGN yields competitive performance on IMDB-TEXT (84.43% accuracy) and IMDB-GRAPH (73.40% cross-validation accuracy), outperforming standard graph neural network baselines without manual architecture tuning.
Limitations and Future Work
- Multiclass Approximation: Multiclass classification was evaluated using an informal one-vs-all scheme with Laplace approximations, leading to worse negative log-likelihood (NLL) calibration on MNIST despite higher raw accuracy. A native multi-class categorical formulation is needed.
- Non-Variational Bound: Unlike variational sparse GPs (e.g., Titsias, Hensman), IGN currently optimizes the marginal log-likelihood directly via mini-batch SGD rather than maximizing an evidence lower bound (ELBO). The authors plan to investigate placing IGN within a fully variational framework to explore deeper hierarchical structures.
Summary updated