Abstract
Gaussian Processes (GPs) are a cornerstone of probabilistic machine learning, offering well-calibrated uncertainty estimates. However, their $O(n^3)$ computational complexity necessitates the use of iterative methods for large-scale applications. While libraries like GPyTorch leverage Conjugate Gradients (CG) and Lanczos algorithms to scale GPs, practitioners often encounter numerical instabilities and poor likelihood performance. This paper demystifies these failures, attributing them to overly relaxed default hyperparameters. We demonstrate that by tightening CG tolerances and increasing Lanczos decomposition ranks, iterative GPs can achieve the stability and accuracy of exact Cholesky-based methods.
Introduction
The scalability of Gaussian Processes has long been a bottleneck in machine learning. Modern iterative solvers provide a theoretical path to $O(n^2)$ or better complexity, but empirical results often show a disconnect between low predictive error (RMSE) and poor marginal likelihood (NLL). This study investigates the root causes of these discrepancies and provides a robust framework for deploying iterative GPs reliably.
Methods and Hyperparameters
We evaluate the impact of three primary hyperparameters that govern the precision of iterative methods:
- CG Tolerance ($\epsilon$): The convergence threshold for linear system solvers.
- Preconditioner Rank ($w$): The rank of the partial Cholesky preconditioner used to accelerate CG.
- Lanczos Decomposition Rank ($k$): The size of the root decomposition used for log-determinant and predictive variance approximations.
Additionally, we re-examine the choice of optimizer, comparing the standard Adam optimizer with the second-order L-BFGS-B method. While L-BFGS-B is traditionally preferred for GPs, it is often avoided in iterative settings due to its sensitivity to numerical noise.
Key Insights: The NLL Trap and Numerical Stability
A critical observation is the "NLL Non-Monotonicity Trap." In iterative GPs, increasing the Lanczos rank $k$ monotonically decreases the predicted posterior variance. However, if the model is overfit, this reduction in variance can lead to a sharp increase in NLL as the model becomes "confidently wrong."
Furthermore, we find that the default CG tolerances in many popular libraries (e.g., $\epsilon=1.0$ during training) are insufficient for maintaining the numerical stability required for hyperparameter optimization. Tightening these tolerances is essential for the successful application of second-order optimizers like L-BFGS-B.
Results and Recommendations
Our experiments across UCI regression benchmarks show that iterative GPs can match exact methods when properly configured. We provide the following "prescription" for practitioners:
- For Training: Use a CG tolerance of $\epsilon = 10^{-3}$ and a preconditioner rank of $w=50$ to ensure stable gradients.
- For Prediction: Use a CG tolerance of $\epsilon = 10^{-2}$ and a large root decomposition size ($k \ge 5000$) to ensure accurate likelihoods.
When these guidelines are followed, L-BFGS-B emerges as a highly effective optimizer, reaching optimal marginal likelihoods in significantly fewer iterations than Adam, provided the underlying linear solves are sufficiently precise.
Conclusion
Iterative Gaussian Processes are not inherently unstable; rather, their reliability is a function of numerical precision. By adopting stricter tolerances and higher-rank approximations, practitioners can leverage the scalability of iterative methods without sacrificing the mathematical rigor of exact GPs. Future work will focus on reducing the per-epoch wall-clock time of second-order methods to further enhance the efficiency of this approach.
Summary updated