Introduction
Deep Gaussian Processes (DGPs) represent a powerful synthesis of deep learning's hierarchical feature extraction and the robust uncertainty quantification inherent in Bayesian Gaussian Processes. Despite their theoretical appeal, DGPs are notoriously difficult to train. Exact inference is intractable, and standard variational approximations require optimizing a large set of auxiliary variables known as inducing inputs across all layers. This paper introduces Subset-of-Data Variational Inference (SOD-VI), a method that simplifies the DGP training process by fixing inducing points to a subset of the training data, thereby reducing the optimization burden and enhancing model stability.
The Challenge of Inducing Point Optimization
Gaussian Processes (GPs) are non-parametric models that provide excellent uncertainty estimates but scale poorly with data size. Sparse approximations mitigate this by using a small set of $M$ inducing points to represent the full dataset. In the context of Deep GPs, where GPs are stacked in layers, the complexity multiplies. Traditional methods require learning the locations and values of inducing inputs for every layer. This leads to a high-dimensional, non-convex optimization landscape that is prone to poor local optima and training instability.
The authors propose a simpler alternative: instead of treating inducing point locations as learnable parameters, why not use a fixed subset of the actual training data? This approach, termed Subset-of-Data Variational Inference (SOD-VI), leverages the information already present in the data to anchor the model's layers.
Subset-of-Data Variational Inference (SOD-VI)
The core innovation of SOD-VI is the selection of a fixed subset $S$ of the training data to serve as inducing inputs for the initial layer. This choice has several profound implications for the model architecture and training objective:
- Fixed Inducing Locations: By fixing the inducing input locations (selected via K-means clustering), the model avoids the overhead of optimizing $M \times D$ parameters per layer.
- Label Integration: Since inducing points are real data points, they possess associated target labels ($y_S$). These labels are incorporated directly into the variational posterior, simplifying the Evidence Lower Bound (ELBO) objective.
- Forward Sampling in Hidden Layers: For intermediate layers, rather than learning new inducing locations, the model samples them from the variational distribution of the preceding layer. This "forward sampling" approach drastically reduces the total number of trainable parameters.
Experimental Evaluation
The authors evaluated SOD-VI on seven standard UCI regression benchmarks, ranging from small datasets like Boston Housing to large ones like the Year Prediction dataset. The performance was compared against several established methods, including standard Sparse GPs (SVGP), Doubly Stochastic Variational Inference (DSVI-DGP), and Stochastic Gradient Hamiltonian Monte Carlo (SGHMC-DGP).
Key Results
- Predictive Accuracy: SOD-VI achieved the lowest Negative Log Predictive Probability (NLPP) on six out of the seven datasets. This metric is particularly important as it accounts for both the accuracy of the mean prediction and the quality of the uncertainty estimate.
- Efficiency and Stability: The reduction in trainable parameters led to more stable optimization. The model was less sensitive to initialization and less likely to converge to suboptimal solutions compared to the more complex DSVI-DGP.
- Depth Performance: Across various depths (from 1 to 4 layers), SOD-VI consistently maintained competitive Root Mean Square Error (RMSE) scores, demonstrating that the simplified approach does not sacrifice the benefits of deep architectures.
Conclusion and Future Directions
Subset-of-Data Variational Inference offers a compelling path forward for making Deep Gaussian Processes more practical and accessible. By replacing complex inducing point optimization with a simple data-driven selection strategy, SOD-VI achieves state-of-the-art performance with a significantly reduced parameter count.
Future work could extend this framework to classification tasks and explore more sophisticated subset selection methods, such as those based on determinantal point processes or leverage scores. Additionally, investigating dynamic subset selection during training could further enhance the model's scalability for massive, streaming datasets.
Summary updated