Identifiability and Sensitivity Analysis of Kriging Weights for the Matern Kernel

Amanda Muyskens, Benjamin W. Priest, Imene Goumiri, M. SchneiderarXiv 2024 · DOI · arXiv

Abstract

Gaussian process (GP) models are effective non-linear models for numerous scientific applications. However, computation of their hyperparameters can be difficult when there is a large number of training observations ($n$) due to the $O(n^3)$ cost of evaluating the likelihood function. Furthermore, non-identifiable hyperparameter values can induce difficulty in parameter estimation. Because of this, maximum likelihood estimation or Bayesian calibration is sometimes omitted and the hyperparameters are estimated with prediction-based methods such as a grid search using cross validation. Kriging, or prediction using a Gaussian process model, amounts to a weighted mean of the data, where training data close to the prediction location as determined by the form and hyperparameters of the kernel matrix are more highly weighted. Our analysis focuses on examination of the commonly utilized Matérn covariance function, of which the radial basis function (RBF) kernel function is the infinity limit of the smoothness parameter. We first perform a collinearity analysis to motivate identifiability issues between the parameters of the Matérn covariance function. We also demonstrate which of its parameters can be estimated using only the predictions. Considering the kriging weights for a fixed training data and prediction location as a function of the hyperparameters, we evaluate their sensitivities - as well as those of the predicted variance - with respect to said hyperparameters. We demonstrate the smoothness parameter $\nu$ is the most sensitive parameter in determining the kriging weights, particularly when the nugget parameter is small, indicating this is the most important parameter to estimate. Finally, we demonstrate the impact of our conclusions on performance and accuracy in a classification problem using a latent Gaussian process model with the hyperparameters selected via a grid search.

Introduction

Gaussian Process (GP) models are powerful tools for non-linear regression and classification, but they come with a significant "hyperparameter headache." Estimating parameters like range ($\rho$), smoothness ($\nu$), and noise (nugget $\tau^2$) is computationally expensive ($O(n^3)$) and often plagued by identifiability issues. Identifiability means that different combinations of parameters can produce nearly identical models, making the optimization surface "flat" and difficult to navigate. This paper investigates which of these parameters actually matter for the final output—the kriging weights (the importance assigned to each training point during prediction)—rather than just the statistical likelihood.

Methodology

The authors employ several sophisticated diagnostic tools to dissect the Matérn kernel:

  • Reparameterization: They show that for predictions, the individual values of process variance ($\sigma^2$) and nugget ($\tau^2$) are non-identifiable. Only their ratio, $\omega^2 = \tau^2/\sigma^2$, affects the kriging weights.
  • Collinearity Analysis: Using a collinearity index ($\gamma$), they map out regions of the parameter space where $\nu$ and $\rho$ become indistinguishable.
  • Global Sensitivity Analysis (BASS): They use Bayesian Adaptive Smoothing Splines (BASS) to decompose the variance of the kriging weights. This allows them to quantify exactly how much of the change in a prediction is caused by varying each hyperparameter.
  • Grid Search Optimization: They compare the performance of full hyperparameter tuning against a simplified strategy of tuning only the most sensitive parameter ($\nu$).

Key Insights and Results

Functional Identifiability

A critical finding is that while two parameter sets might look different in a correlation plot, they can result in nearly identical kriging weights. This "functional identifiability" suggests that the model's predictive behavior is less sensitive to certain parameter variations than the likelihood function might imply.

Comparison of correlation functions vs. kriging weights

The Collinearity Trap and Sensitivity

As smoothness increases, the parameters $\nu$ and $\rho$ become highly collinear, meaning tuning both is often redundant and computationally wasteful. In 1-D simulations, the smoothness parameter $\nu$ accounted for a significant portion of the sensitivity in kriging weights (up to ~31%), far outweighing the range parameter in many scenarios. This suggests that in GP fitting convention, where researchers often fix $\nu$ and tune $\rho$, the opposite approach might be more effective: $\nu$ is often the most sensitive parameter for determining kriging weights, especially when noise is low.

Comprehensive analysis of correlation, kriging weights, and parameter collinearity maps

Performance in Classification

In a real-world star-galaxy classification task using a latent GP model, estimating only $\nu$ while fixing other parameters led to a 91x speedup compared to estimating all parameters for a sample size of $n=2000$. For larger training sets, there was no statistically significant difference in classification accuracy between the full tuning approach and the simplified "tune only $\nu$" strategy.

Conclusion

This study reveals that the smoothness parameter ($\nu$) is the most critical hyperparameter for Gaussian Process predictions using the Matérn kernel. By identifying which parameters truly drive prediction weights, researchers can fix redundant parameters and tune only the most sensitive ones, achieving massive speedups in training without sacrificing accuracy. Future work could explore anisotropic kernels or extend this analysis to Neural Tangent Kernels (NTK) to see if similar sensitivity patterns emerge in deep learning contexts.

Summary updated

Abstract

Gaussian process (GP) models are effective non-linear models for numerous scientific applications. However, computation of their hyperparameters can be difficult when there is a large number of training observations (n) due to the O(n^3) cost of evaluating the likelihood function. Furthermore, non-identifiable hyperparameter values can induce difficulty in parameter estimation. Because of this, maximum likelihood estimation or Bayesian calibration is sometimes omitted and the hyperparameters are estimated with prediction-based methods such as a grid search using cross validation. Kriging, or prediction using a Gaussian process model, amounts to a weighted mean of the data, where training data close to the prediction location as determined by the form and hyperparameters of the kernel matrix are more highly weighted. Our analysis focuses on examination of the commonly utilized Matern covariance function, of which the radial basis function (RBF) kernel function is the infinity limit of the smoothness parameter. We first perform a collinearity analysis to motivate identifiability issues between the parameters of the Matern covariance function. We also demonstrate which of its parameters can be estimated using only the predictions. Considering the kriging weights for a fixed training data and prediction location as a function of the hyperparameters, we evaluate their sensitivities - as well as those of the predicted variance - with respect to said hyperparameters. We demonstrate the smoothness parameter nu is the most sensitive parameter in determining the kriging weights, particularly when the nugget parameter is small, indicating this is the most important parameter to estimate. Finally, we demonstrate the impact of our conclusions on performance and accuracy in a classification problem using a latent Gaussian process model with the hyperparameters selected via a grid search.
BibTeX
@misc{muyskens2024identifiability,
  author = {Amanda Muyskens and Benjamin W. Priest and Imene Goumiri and M. Schneider},
  title = {{Identifiability and Sensitivity Analysis of Kriging Weights for the Matern Kernel}},
  year = {2024},
  eprint = {2410.08310},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.08310},
}
Figure