Abstract
This paper investigates the speed of convergence of the learning curve in Gaussian process (GP) regression. The learning curve, defined as the integral of the mean squared error (MSE) over the input parameter space, characterizes the average generalization error of the surrogate model. We present a rigorous proof of a theorem that determines the MSE as a function of the number of observations for a broad class of kernels across any dimension. This result generalizes previous findings limited to specific kernels or low-dimensional spaces. Furthermore, we demonstrate how these asymptotic behaviors can be leveraged to optimize resource allocation, validated through a nuclear safety application involving stochastic neutron transport simulations.
Introduction
Gaussian Process regression is a cornerstone of modern surrogate modeling, particularly when approximating computationally expensive computer simulations. In scenarios involving noisy data—such as stochastic simulators or physical experiments—researchers face a fundamental optimization problem: the trade-off between sampling many points with low precision versus fewer points with high precision.
To navigate this trade-off, one must understand the "learning curve," which tracks the reduction of the Integrated Mean Squared Error (IMSE) as the observation budget increases. While previous literature established convergence rates for simple 1D cases or degenerate kernels with finite features, a general theory for the infinite-dimensional kernels common in machine learning (e.g., Matèrn or Gaussian kernels) has remained elusive. This research fills that gap by linking error reduction directly to the spectral properties of the covariance kernel.
Theoretical Framework and Methodology
The study utilizes Mercer’s Theorem to decompose the GP’s covariance kernel into its constituent eigenvalues ($\lambda_p$) and eigenfunctions ($\phi_p$). This decomposition allows the GP to be analyzed as an infinite sum of independent components.
The central contribution is a theorem proving that as the number of observations $n$ becomes sufficiently large, the MSE at any point $x$ converges according to: $$\sigma^2(x) \approx \sum_{p \ge 0} \frac{\tau \lambda_p}{\tau + \lambda_p} \phi_p(x)^2$$ where $\tau$ represents the noise level.
To establish this, the authors employ a "sandwich" proof technique:
- Lower Bound: Utilizing the Karhunen-Loève decomposition to establish a minimum error threshold.
- Upper Bound: Constructing a specific Linear Unbiased Predictor (LUP) and applying the Woodbury-Sherman-Morrison formula to demonstrate that the error does not exceed the predicted threshold.
Key Insights: Regularity and Convergence
The research highlights that the "smoothness" or regularity of the kernel is the primary driver of learning speed. For a 1D Matèrn kernel with regularity $\nu$, the error decays at a rate of $\tau^{1 - 1/2\nu}$. Consequently, smoother processes (higher $\nu$) exhibit significantly faster convergence.
Crucially, the paper notes that while simple models often follow the classic Monte Carlo error drop rate of $1/n$, non-degenerate kernels typically converge much more slowly. This explains why standard budget-planning heuristics often fail when applied to complex Gaussian Process models. By understanding these rates, engineers can derive optimal resource allocation strategies. In cases of heterogeneous noise, the theory suggests allocating more repetitions to noisier regions or areas where the IMSE is locally concentrated.
Industrial Application: Nuclear Safety
The theoretical findings were validated using both synthetic benchmarks (Fractional Brownian Motion, d-Dimensional Matèrn, and Gaussian kernels) and an industrial case study involving the MORET code. The MORET code is a stochastic neutron transport simulator used to evaluate the safety of dry plutonium storage.
In the industrial application, the objective was to reach a target error of $2 \cdot 10^{-4}$. The proposed theory predicted a required budget of $s=36$ repetitions per point. In practice, the target was achieved at $s=31$, confirming that the theory provides a reliable, slightly conservative estimate for real-world engineering requirements. Furthermore, the "Optimal Allocation" strategy derived from the theorem consistently outperformed uniform sampling, achieving lower error for the same total computational expenditure.
Conclusion and Future Work
This work provides a rigorous mathematical foundation for predicting the performance of Gaussian Process regression across diverse kernels and dimensions. While the current proof for optimal resource allocation assumes a diagonal covariance matrix, numerical results suggest the heuristic remains highly effective in general cases. Future research may focus on formalizing proofs for non-diagonal cases and exploring how these convergence rates evolve when design points are selected via active learning or sequential design rather than uniform sampling.
Summary updated