Calibrated Computation-Aware Gaussian Processes

Disha Hegde, Mohamed Adil, Jon CockayneAISTATS 2025

Gaussian Processes (GPs) represent a cornerstone of non-parametric Bayesian regression, offering robust uncertainty quantification alongside predictions. However, their $O(d^3)$ computational complexity limits their application to large-scale datasets. Computation-Aware GPs (CAGPs) mitigate this by employing iterative probabilistic linear solvers (PLS) to approximate matrix inversions. While efficient, the standard CAGP-CG (based on Conjugate Gradients) often suffers from poor calibration, yielding overly conservative uncertainty estimates. This paper addresses this gap by establishing a theoretical link between solver calibration and GP model reliability.

The Heredity of Calibration

A primary contribution of this work is the rigorous proof that "calibratedness" is a hereditary property. Specifically, if a PLS is "strongly calibrated"—meaning its output distribution accurately reflects its internal approximation error—the resulting CAGP model will also be calibrated.

To leverage this, the authors introduce CAGP-GS, a framework utilizing Probabilistic Stationary Iterative Methods (PSIMs), specifically the Gauss-Seidel (GS) method. Unlike Conjugate Gradients, the GS approach allows for better-calibrated approximations. The framework also incorporates a marginalization technique to account for computational uncertainty arising from early termination, ensuring the model "knows what it doesn't know."

Experimental Evaluation

The authors evaluated CAGP-GS against the state-of-the-art CAGP-CG across synthetic data, UCI benchmarks, and a massive ERA5 global temperature dataset containing approximately 1 million points.

Synthetic and Theoretical Validation

Using a Matérn 3/2 covariance function, the authors demonstrated that CAGP-GS achieves significantly better Negative Log-Likelihood (NLL) scores compared to CAGP-CG. NLL rewards models for being accurate about their own uncertainty; lower scores indicate superior calibration.

NLL performance on synthetic data across different lengthscales (l). Lower NLL indicates better-calibrated uncertainty.

In calibration tests, CAGP-GS yielded Kolmogorov-Smirnov (KS) test p-values (e.g., 0.6689 and 0.4707) indicating high reliability. In contrast, CAGP-CG p-values approached zero, confirming its poor uncertainty quantification.

Real-World Benchmarks

On UCI datasets such as Concrete Compressive Strength, Bike Sharing, and 3D Road Network, CAGP-GS demonstrated superior initial convergence and more stable error reduction.

RMSE and NLL convergence on UCI benchmark datasets. CAGP-GS (blue) shows smoother and faster initial error reduction compared to CAGP-CG (orange).

As shown in the benchmarks, CAGP-GS consistently achieves lower Root Mean Square Error (RMSE) in early iterations and maintains better NLL throughout the process. This "smoothing" effect of the Gauss-Seidel method makes it particularly valuable for computationally expensive tasks where only a few iterations are feasible.

Discussion and Limitations

While CAGP-GS offers superior calibration and early-stage convergence, it faces specific scalability challenges. The computational cost of CAGP-GS scales more steeply with the number of test points ($d_{test}$) compared to CAGP-CG, making it ideal for scenarios with many training points but fewer prediction targets. Furthermore, the sequential nature of Gauss-Seidel iterations makes it less amenable to GPU acceleration than the parallelizable Conjugate Gradient method.

Future research may explore hybrid models that combine the initial smoothing and calibration of Gauss-Seidel with the asymptotic speed and parallelizability of Conjugate Gradients, potentially offering a "best of both worlds" solution for large-scale Bayesian regression.

Summary updated

BibTeX
@inproceedings{hegde2025calibrated,
  author = {Disha Hegde and Mohamed Adil and Jon Cockayne},
  title = {{Calibrated Computation-Aware Gaussian Processes}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2025},
}
Figure