Abstract
Recent advances in Deep Gaussian Processes (DGPs) show the potential to have more expressive representation than that of traditional Gaussian Processes (GPs). However, there exists a pathology of deep Gaussian processes that their learning capacities reduce significantly when the number of layers increases. In this paper, we present a new analysis in DGPs by studying its corresponding nonlinear dynamic systems to explain the issue. Existing work reports the pathology for the squared exponential kernel function. We extend our investigation to four types of common stationary kernel functions. The recurrence relations between layers are analytically derived, providing a tighter bound and the rate of convergence of the dynamic systems. We demonstrate our finding with a number of experimental results.
Introduction
Deep Gaussian Processes (DGPs) are hierarchical models that stack multiple GP layers to enhance representational power. However, as depth increases, these models often suffer from a "pathology" where the resulting functions become increasingly flat, losing their ability to distinguish between different inputs. This collapse in learning capacity has historically limited the practical application of very deep GPs. This paper provides a mathematical diagnosis of this phenomenon by framing the layer-to-layer transition as a nonlinear dynamical system.
Methodology: DGPs as Dynamical Systems
The authors analyze the evolution of the squared Euclidean distance $Z_n$ between two inputs as they propagate through $n$ layers. This progression is modeled as a nonlinear recurrence relation: $\mathbb{E}[Z_n] = h(\mathbb{E}[Z_{n-1}]; \theta)$, where $\theta$ represents the kernel hyperparameters.
By deriving specific recurrence relations for five popular kernels—Squared Exponential (SE), Cosine (COS), Spectral Mixture (SM), Rational Quadratic (RQ), and Periodic (PER)—the researchers can analytically determine the stability of the DGP.
Key Insights: Bifurcation and Stability
Using tools from chaos theory, the paper employs bifurcation plots to visualize the "steady state" of the DGP. If the distance $Z_n$ converges to zero, the model becomes pathological (flat). Conversely, if it converges to a non-zero value or enters a chaotic regime, the model remains expressive.
A significant discovery is that Spectral Mixture (SM) kernels are naturally immune to pathology due to their unique mathematical structure. For other kernels, the authors identify specific "safe" hyperparameter zones that prevent functional collapse. This provides a tighter bound on model behavior than previous exponential approximations.
Empirical Results
The theoretical findings were validated across several benchmarks:
- Numerical Accuracy: The derived recurrence relations provided near-perfect fits to empirical data, significantly outperforming previous theoretical bounds.
- MNIST Classification: A standard DGP that failed with ~10% accuracy (random guessing) was improved to 91.21% accuracy simply by constraining hyperparameters to the identified "safe" regions.
- Regression Tasks: On the Boston Housing and Diabetes datasets, the model's RMSE was shown to correlate directly with the stability of the derived recurrence trajectories.
Conclusion
By characterizing the depth of a neural network as "time" in a dynamical system, this research provides a rigorous framework for designing stable Deep Gaussian Processes. While this approach successfully identifies safe hyperparameter regimes, future work is needed to explore the optimization landscapes within these zones and the interaction with complex mean functions.
Summary updated