When are Iterative Gaussian Processes Reliably Accurate?

Wesley J. Maddox, Sanyam Kapoor, Andrew Gordon Wilson

Abstract

Gaussian Processes (GPs) are a cornerstone of probabilistic machine learning, offering well-calibrated uncertainty estimates. However, their $O(n^3)$ computational complexity necessitates the use of iterative methods for large-scale applications. While libraries like GPyTorch leverage Conjugate Gradients (CG) and Lanczos algorithms to scale GPs, practitioners often encounter numerical instabilities and poor likelihood performance. This paper demystifies these failures, attributing them to overly relaxed default hyperparameters. We demonstrate that by tightening CG tolerances and increasing Lanczos decomposition ranks, iterative GPs can achieve the stability and accuracy of exact Cholesky-based methods.

Introduction

The scalability of Gaussian Processes has long been a bottleneck in machine learning. Modern iterative solvers provide a theoretical path to $O(n^2)$ or better complexity, but empirical results often show a disconnect between low predictive error (RMSE) and poor marginal likelihood (NLL). This study investigates the root causes of these discrepancies and provides a robust framework for deploying iterative GPs reliably.

Methods and Hyperparameters

We evaluate the impact of three primary hyperparameters that govern the precision of iterative methods:

  1. CG Tolerance ($\epsilon$): The convergence threshold for linear system solvers.
  2. Preconditioner Rank ($w$): The rank of the partial Cholesky preconditioner used to accelerate CG.
  3. Lanczos Decomposition Rank ($k$): The size of the root decomposition used for log-determinant and predictive variance approximations.

Additionally, we re-examine the choice of optimizer, comparing the standard Adam optimizer with the second-order L-BFGS-B method. While L-BFGS-B is traditionally preferred for GPs, it is often avoided in iterative settings due to its sensitivity to numerical noise.

Key Insights: The NLL Trap and Numerical Stability

A critical observation is the "NLL Non-Monotonicity Trap." In iterative GPs, increasing the Lanczos rank $k$ monotonically decreases the predicted posterior variance. However, if the model is overfit, this reduction in variance can lead to a sharp increase in NLL as the model becomes "confidently wrong."

The non-monotonic relationship between Negative Log Likelihood (NLL) and variance (sigma), illustrating the risk of becoming 'confidently wrong'.

Furthermore, we find that the default CG tolerances in many popular libraries (e.g., $\epsilon=1.0$ during training) are insufficient for maintaining the numerical stability required for hyperparameter optimization. Tightening these tolerances is essential for the successful application of second-order optimizers like L-BFGS-B.

Results and Recommendations

Our experiments across UCI regression benchmarks show that iterative GPs can match exact methods when properly configured. We provide the following "prescription" for practitioners:

  • For Training: Use a CG tolerance of $\epsilon = 10^{-3}$ and a preconditioner rank of $w=50$ to ensure stable gradients.
  • For Prediction: Use a CG tolerance of $\epsilon = 10^{-2}$ and a large root decomposition size ($k \ge 5000$) to ensure accurate likelihoods.
Empirical evaluation of iterative GP performance. From left to right: optimization stability (Loss and Lengthscale) under varying tolerances, and the impact of root decomposition size and CG tolerance on test metrics.

When these guidelines are followed, L-BFGS-B emerges as a highly effective optimizer, reaching optimal marginal likelihoods in significantly fewer iterations than Adam, provided the underlying linear solves are sufficiently precise.

Conclusion

Iterative Gaussian Processes are not inherently unstable; rather, their reliability is a function of numerical precision. By adopting stricter tolerances and higher-rank approximations, practitioners can leverage the scalability of iterative methods without sacrificing the mathematical rigor of exact GPs. Future work will focus on reducing the per-epoch wall-clock time of second-order methods to further enhance the efficiency of this approach.

Summary updated

Abstract

While recent work on conjugate gradient methods and Lanczos decompositions have achieved scalable Gaussian process inference with highly accurate point predictions, in several implementations these iterative methods appear to struggle with numerical instabilities in learning kernel hyperparameters, and poor test likelihoods. By investigating CG tolerance, preconditioner rank, and Lanczos decomposition rank, we provide a particularly simple prescription to correct these issues: we recommend that one should use a small CG tolerance ($ε\leq 0.01$) and a large root decomposition size ($r \geq 5000$). Moreover, we show that L-BFGS-B is a compelling optimizer for Iterative GPs, achieving convergence with fewer gradient updates.
BibTeX
@misc{maddox2021when,
  author = {Wesley J. Maddox and Sanyam Kapoor and Andrew Gordon Wilson},
  title = {{When are Iterative Gaussian Processes Reliably Accurate?}},
  year = {2021},
  eprint = {2112.15246},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2112.15246},
}
Figure