Regularity dependence of the rate of convergence of the learning curve for Gaussian process regression

Loïc Le Gratiet, Josselin GarnierarXiv 2012 · DOI · arXiv

Abstract

This paper investigates the speed of convergence of the learning curve in Gaussian process (GP) regression. The learning curve, defined as the integral of the mean squared error (MSE) over the input parameter space, characterizes the average generalization error of the surrogate model. We present a rigorous proof of a theorem that determines the MSE as a function of the number of observations for a broad class of kernels across any dimension. This result generalizes previous findings limited to specific kernels or low-dimensional spaces. Furthermore, we demonstrate how these asymptotic behaviors can be leveraged to optimize resource allocation, validated through a nuclear safety application involving stochastic neutron transport simulations.

Introduction

Gaussian Process regression is a cornerstone of modern surrogate modeling, particularly when approximating computationally expensive computer simulations. In scenarios involving noisy data—such as stochastic simulators or physical experiments—researchers face a fundamental optimization problem: the trade-off between sampling many points with low precision versus fewer points with high precision.

To navigate this trade-off, one must understand the "learning curve," which tracks the reduction of the Integrated Mean Squared Error (IMSE) as the observation budget increases. While previous literature established convergence rates for simple 1D cases or degenerate kernels with finite features, a general theory for the infinite-dimensional kernels common in machine learning (e.g., Matèrn or Gaussian kernels) has remained elusive. This research fills that gap by linking error reduction directly to the spectral properties of the covariance kernel.

Theoretical Framework and Methodology

The study utilizes Mercer’s Theorem to decompose the GP’s covariance kernel into its constituent eigenvalues ($\lambda_p$) and eigenfunctions ($\phi_p$). This decomposition allows the GP to be analyzed as an infinite sum of independent components.

The central contribution is a theorem proving that as the number of observations $n$ becomes sufficiently large, the MSE at any point $x$ converges according to: $$\sigma^2(x) \approx \sum_{p \ge 0} \frac{\tau \lambda_p}{\tau + \lambda_p} \phi_p(x)^2$$ where $\tau$ represents the noise level.

To establish this, the authors employ a "sandwich" proof technique:

  1. Lower Bound: Utilizing the Karhunen-Loève decomposition to establish a minimum error threshold.
  2. Upper Bound: Constructing a specific Linear Unbiased Predictor (LUP) and applying the Woodbury-Sherman-Morrison formula to demonstrate that the error does not exceed the predicted threshold.

Key Insights: Regularity and Convergence

The research highlights that the "smoothness" or regularity of the kernel is the primary driver of learning speed. For a 1D Matèrn kernel with regularity $\nu$, the error decays at a rate of $\tau^{1 - 1/2\nu}$. Consequently, smoother processes (higher $\nu$) exhibit significantly faster convergence.

Crucially, the paper notes that while simple models often follow the classic Monte Carlo error drop rate of $1/n$, non-degenerate kernels typically converge much more slowly. This explains why standard budget-planning heuristics often fail when applied to complex Gaussian Process models. By understanding these rates, engineers can derive optimal resource allocation strategies. In cases of heterogeneous noise, the theory suggests allocating more repetitions to noisier regions or areas where the IMSE is locally concentrated.

Industrial Application: Nuclear Safety

The theoretical findings were validated using both synthetic benchmarks (Fractional Brownian Motion, d-Dimensional Matèrn, and Gaussian kernels) and an industrial case study involving the MORET code. The MORET code is a stochastic neutron transport simulator used to evaluate the safety of dry plutonium storage.

In the industrial application, the objective was to reach a target error of $2 \cdot 10^{-4}$. The proposed theory predicted a required budget of $s=36$ repetitions per point. In practice, the target was achieved at $s=31$, confirming that the theory provides a reliable, slightly conservative estimate for real-world engineering requirements. Furthermore, the "Optimal Allocation" strategy derived from the theorem consistently outperformed uniform sampling, achieving lower error for the same total computational expenditure.

Conclusion and Future Work

This work provides a rigorous mathematical foundation for predicting the performance of Gaussian Process regression across diverse kernels and dimensions. While the current proof for optimal resource allocation assumes a diagonal covariance matrix, numerical results suggest the heuristic remains highly effective in general cases. Future research may focus on formalizing proofs for non-diagonal cases and exploring how these convergence rates evolve when design points are selected via active learning or sequential design rather than uniform sampling.

Summary updated

Abstract

This paper deals with the speed of convergence of the learning curve in a Gaussian process regression framework. The learning curve describes the average generalization error of the Gaussian process used for the regression. More specifically, it is defined in this paper as the integral of the mean squared error over the input parameter space with respect to the probability measure of the input parameters. The main result is the proof of a theorem giving the mean squared error in function of the number of observations for a large class of kernels and for any dimension when the number of observations is large. From this result, we can deduce the asymptotic behavior of the generalization error. The presented proof generalizes previous ones that were limited to more specific kernels or to small dimensions (one or two). The result can be used to build an optimal strategy for resources allocation. This strategy is applied successfully to a nuclear safety problem.
BibTeX
@misc{gratiet2012regularity,
  author = {Loïc Le Gratiet and Josselin Garnier},
  title = {{Regularity dependence of the rate of convergence of the learning curve for Gaussian process regression}},
  year = {2012},
  eprint = {1210.2879},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1210.2879},
}

Loading related content...

Figure