Subset-of-Data Variational Inference for Deep Gaussian-Processes Regression

Ayush Jain, P. K. Srijith, Mohammad Emtiyaz Khan

Introduction

Deep Gaussian Processes (DGPs) represent a powerful class of models that stack multiple layers of Gaussian Processes to capture complex, non-linear functional relationships. While they offer superior uncertainty quantification compared to standard neural networks, training them is notoriously difficult. Scalability in DGPs typically relies on "sparse approximations" using inducing points—a small set of representative data points that summarize the entire dataset. Traditionally, researchers must optimize both the locations and the values of these inducing points across every layer, resulting in a massive number of parameters and a highly unstable optimization landscape prone to poor local optima.

The Subset-of-Data (SoD) Approach

This paper introduces Subset-of-Data Variational Inference (SoD-VI), a method that simplifies DGP training by fundamentally changing how inducing points are handled. Instead of treating inducing point locations as free parameters to be learned, the authors propose using a fixed subset of the actual training data (the "Subset-of-Data") as inducing points.

The Subset-of-Data (SoD) method uses a fixed subset of training data to anchor the variational posterior across layers.

By fixing these locations ($Z$), the number of trainable parameters is drastically reduced compared to standard methods like Doubly Stochastic Variational Inference (DSVI). The subset can be selected using simple, efficient techniques such as K-means clustering to ensure it remains representative of the overall data distribution.

Key Technical Innovations

The SoD-VI framework leverages the properties of the training data to improve inference:

  • Label Exploitation: Because the inducing points are real training samples, their ground-truth labels ($y_S$) are known. The model incorporates these labels directly into the variational posterior, "anchoring" the approximation and simplifying the Evidence Lower Bound (ELBO) calculation.
  • Sampling vs. Optimization: In deeper layers, rather than optimizing inducing inputs, the model samples them from the conditional variational distribution. This maintains the hierarchical structure of the DGP without adding more parameters to the gradient descent process.
  • Stability through Simplicity: Reducing the number of "moving parts" makes the training process more stable and less sensitive to initialization, addressing a major pain point in DGP research.

Experimental Results

The authors evaluated SoD-DGP on seven standard UCI regression benchmarks, including small-scale datasets like Boston Housing and large-scale ones like the Year Prediction MSD (over 500,000 samples).

The proposed method demonstrated superior predictive accuracy, achieving the lowest Negative Log Predictive Probability (NLPP) on nearly all datasets. This indicates that SoD-DGP is not only more accurate but also more confident in its correct predictions. A standout result was achieved on the Naval Propulsion dataset, where SoD-DGP1 reached an NLPP of -8.15, significantly outperforming the next best baseline (SVGP at -7.42).

Conclusion

The "Subset-of-Data" approach demonstrates that the traditional practice of optimizing inducing point locations may be unnecessary for Deep Gaussian Processes. By using real data points as anchors, SoD-VI achieves state-of-the-art performance with a simpler and more efficient training process. While currently focused on regression, future work aims to extend this method to classification and explore more sophisticated subset selection schemes like leverage scores or Determinantal Point Processes (DPPs).

Summary updated

BibTeX
@inproceedings{jain2021subset,
  author = {Ayush Jain and P. K. Srijith and Mohammad Emtiyaz Khan},
  title = {{Subset-of-Data Variational Inference for Deep Gaussian-Processes Regression}},
  booktitle = {Conference on Uncertainty in Artificial Intelligence},
  year = {2021},
  url = {https://openreview.net/forum?id=7yWCqEwyK4P},
}
Figure