Introduction
Deep Gaussian Processes (DGPs) represent a powerful class of hierarchical Bayesian models, combining the flexible feature extraction of deep learning with the principled uncertainty quantification of Gaussian Processes. However, training DGPs remains a significant challenge. Standard variational inference techniques often suffer from "posterior collapse," where latent layers lose their stochasticity and revert to deterministic functions. Conversely, Deep Kernel Learning (DKL)—which uses a deterministic neural network to warp inputs for a standard GP—is highly scalable but lacks Bayesian uncertainty in its feature extractor, making it prone to severe overfitting.
This paper introduces Conditional Deep Gaussian Processes, a novel framework that bridges the gap between fully Bayesian DGPs and deterministic DKL. By conditioning intermediate GP layers on optimizable "hyperdata," the authors enable scalable empirical Bayes training that preserves uncertainty propagation while avoiding the pitfalls of variational collapse.
Methodology
The core innovation of this work is the introduction of an intermediate conditioning layer termed "hyperdata."
Hyperdata and Neural Network Parameterization
In a standard DGP, intermediate layers are typically zero-mean GPs. The authors instead condition these hidden layers on a set of support points, or hyperdata. These points act similarly to inducing points in sparse GPs but are treated as optimizable hyperparameters. To leverage the representational power of deep learning, the hyperdata values are parameterized by a neural network: $u = \text{NN}_w(z)$. This allows the model to learn complex, data-driven support for the intermediate mappings.
Moment Matching and Empirical Bayes
Marginalizing out the intermediate GP layers in a DGP usually results in an intractable, non-Gaussian prior. To maintain tractability, the authors employ a moment-matching technique. They approximate the complex marginal prior with a Gaussian distribution that matches its exact mean and covariance.
With this Gaussian approximation, the model can be trained via empirical Bayes by maximizing the approximate marginal likelihood (logML). This process jointly optimizes the GP hyperparameters and the neural network weights that define the hyperdata, providing a unified training objective that avoids the need for complex variational bounds.
Experimental Evaluation
The authors evaluated the Conditional DGP on two benchmark time-series datasets known for their challenging extrapolation requirements.
Mauna Loa CO2 Data
The Mauna Loa dataset requires capturing both long-term trends and seasonal periodicity. A 2-layer Conditional DGP successfully extrapolated high-frequency signals, achieving a log marginal likelihood of 338. A 3-layer variant (logML 253) effectively captured the long-term trend while maintaining predictive uncertainty that enveloped the test data. In comparison, variational DGPs struggled significantly, yielding much lower Evidence Lower Bound (ELBO) scores (approx. 127-135) and failing to extrapolate as effectively.
Airline Passenger Data
This dataset features non-stationary periodic trends with variance that increases over time. The 2-layer Conditional DGP (logML 28.5) successfully extrapolated these oscillating trends using only simple base kernels (e.g., Squared Exponential). In contrast, the variational DGP baseline suffered from vanishing uncertainty in the latent layers, causing its predictive mean to collapse toward zero (logML 7.25).
Discussion and Key Insights
The Conditional DGP framework offers several theoretical and practical advantages:
- Bridge to Deep Kernel Learning: The authors demonstrate that in the limit of infinitely dense hyperdata, the Conditional DGP reduces to standard DKL. With sparse hyperdata, the model acts as an ensemble of deep kernels, preserving the uncertainty that DKL discards.
- Emergent Expressivity: One of the most striking findings is that the hierarchical architecture naturally generates complex kernels. Simple base kernels can extrapolate intricate periodic patterns without requiring manually engineered periodic kernels.
- Robustness to Collapse: By using moment matching and marginal likelihood maximization rather than variational inference, the model avoids the "posterior collapse" common in deeper Bayesian architectures.
Conclusion and Future Work
While the Conditional DGP provides a robust middle ground for hierarchical GP modeling, it is not without limitations. The moment-matching approximation forces the marginal prior to be Gaussian, potentially losing information from the heavy-tailed statistics of the true DGP prior. Additionally, treating neural network weights as point estimates via empirical Bayes still carries some risk of overfitting.
Future research directions include upgrading hyperdata from deterministic parameters to full random variables. Applying moment matching to approximate the marginal prior over these variables could lead to a more fully Bayesian inference scheme while maintaining the scalability and stability demonstrated in this work.
Summary updated