Abstract
Recent advances in Deep Gaussian Processes (DGPs) show the potential to have more expressive representation than that of traditional Gaussian Processes (GPs). However, there exists a pathology of deep Gaussian processes that their learning capacities reduce significantly when the number of layers increases. In this paper, we present a new analysis in DGPs by studying its corresponding nonlinear dynamic systems to explain the issue. Existing work reports the pathology for the squared exponential kernel function. We extend our investigation to four types of common stationary kernel functions. The recurrence relations between layers are analytically derived, providing a tighter bound and the rate of convergence of the dynamic systems. We demonstrate our finding with a number of experimental results.
Introduction
Deep Gaussian Processes (DGPs) are hierarchical compositions of Gaussian Processes designed to overcome the limitations of shallow models. Theoretically, they should offer superior representational power. However, practitioners have long encountered a "pathology": as depth increases, the functions generated by DGP priors tend to become flat and lose their ability to distinguish between different inputs. This paper provides a unified theoretical framework using nonlinear dynamics to analyze this collapse across various kernel types.
Methods: DGPs as Nonlinear Recurrence Systems
The core of the analysis treats the propagation of data through DGP layers as a nonlinear recurrence system. The authors focus on the evolution of $Z_n$, the squared Euclidean distance between two outputs at layer $n$.
By utilizing Moment-Generating Functions (MGFs), the researchers derived analytical recurrence relations for five major kernels: Squared Exponential (SE), Cosine (COS), Periodic (PER), Rational Quadratic (RQ), and Spectral Mixture (SM). This approach allows for the use of chaos theory tools, such as bifurcation plots, to identify stable "safe" regions versus pathological regimes where distances converge to zero.
Key Insights and Theoretical Findings
A pivotal discovery is that Spectral Mixture (SM) kernels are inherently immune to the pathology; unlike SE kernels, the distance between points in an SM-based DGP does not collapse as depth increases. Furthermore, the authors provide the first analytical explanation for why non-zero mean functions mitigate the issue: they introduce a "non-centrality" parameter in the underlying Chi-squared distribution of distances, preventing the system from reaching a zero-value fixed point.
For kernels susceptible to collapse, the authors identified specific hyperparameter thresholds (e.g., $\sigma^2/\ell^2 > 1/m$ for SE kernels) that act as tipping points. Staying within these "safe zones" via constrained regularization ensures the model maintains its representational capacity during training.
Experimental Results
The theoretical framework was validated on several benchmarks:
- MNIST Classification: Standard zero-mean DGPs typically fail, achieving only ~10% accuracy. By applying the proposed hyperparameter constraints, accuracy jumped to 91.21%.
- Loss Landscape Analysis: Visualization revealed that standard DGPs often fall into "unsafe" flat regions during optimization. The constrained versions maintain a more navigable surface, facilitating better convergence.
- Convergence Rates: The study showed that different kernels converge at varying rates, with SM kernels being the fastest and Rational Quadratic (RQ) kernels the slowest.
Conclusion
This work provides a rigorous foundation for understanding and fixing the depth-induced pathology in DGPs. While the 91% accuracy on MNIST is a significant leap for zero-mean models, it still trails state-of-the-art DGPs using complex mean functions. Future research will explore automated kernel discovery and further optimization of the loss landscape in extremely deep models.
Summary updated