TL;DR
This paper investigates the practical challenges of using Kriging models for analyzing deterministic computer experiments, especially when dealing with limited data. The authors highlight the unreliability of standard Maximum Likelihood Estimation (MLE) for covariance parameters with small sample sizes and the critical impact of trend misspecification, particularly in extrapolation scenarios. They propose a novel approach combining additive models with Kriging to better handle non-linear trends and improve predictive accuracy.
What Problem Does This Paper Address?
Computer experiments, which simulate complex systems, often generate large amounts of data. Kriging is a powerful statistical technique used to build surrogate models (metamodels) for these expensive simulations, allowing for efficient exploration and optimization. However, in practice, the design of experiments (the set of input points where simulations are run) can be small relative to the dimensionality of the input space. This paper addresses two key issues that arise in such scenarios:
- Reliability of Parameter Estimation: Standard methods for estimating the parameters of the Kriging model (specifically, the covariance parameters) often rely on Maximum Likelihood Estimation (MLE). The paper questions the reliability of MLE when the number of data points is small, as it can lead to biased or highly variable estimates.
- Impact of Trend Misspecification: Kriging models assume a deterministic trend component and a random Gaussian process component. Choosing the wrong trend function (e.g., assuming a linear trend when the true trend is non-linear) can significantly degrade the model's predictive performance, especially when extrapolating beyond the observed data. This is a common situation in high-dimensional problems.
The paper aims to provide practical insights and propose a more robust methodology for these challenging situations.
Methods
The paper explores several methodological aspects:
- Kriging Fundamentals: It starts by recalling the basic principles of Kriging, including its formulation as a linear optimal predictor and the general decomposition of the response as a sum of a deterministic trend and a Gaussian process.
- Maximum Likelihood Estimation (MLE) Analysis: The authors conduct simulation studies to evaluate the performance of MLE for estimating covariance parameters. They analyze the bias and variance of the estimators and compare them to theoretical asymptotic results derived from the Fisher Information Matrix.
- Trend Misspecification Analysis: Through toy experiments, the paper demonstrates the consequences of using incorrect trend functions (Ordinary Kriging vs. Universal Kriging with linear or quadratic trends) on different experimental designs.
- Additive Models (AMs) and Kriging Combination: A novel approach is proposed that combines the flexibility of Additive Models (AMs) with the interpolation capabilities of Kriging. AMs allow for flexible modeling of non-linear trends by summing univariate functions of each input variable. The proposed method uses a two-step approach:
- Estimate the additive trend using a backfitting algorithm.
- Apply Simple Kriging to the residuals (the difference between the observed data and the estimated trend), estimating the Kriging covariance parameters from these residuals.
- Sequential Estimation Technique: A refined two-step approach is introduced where the trend is fitted on one subset of data (X1) and the covariance parameters are estimated on a separate validation set (X2). This aims to provide more robust covariance parameter estimates, especially when the trend is non-linear.
Datasets
The paper uses simulated datasets for its experimental studies:
- Toy Experiments: Monodimensional Gaussian Processes with and without linear/quadratic trends are simulated to test the MLE of covariance parameters and the impact of trend misspecification.
- 3D Application (Porous Media): A 3-dimensional example from an industrial test case (porous media simulation) is used to demonstrate the proposed Kriging with Additive Trend (KAT) method. Data is generated from a flow simulator. The experiments involve different designs:
- X1: A 20-element Hammersley sequence.
- X2: A 14-point D-optimal design.
- F: An 11-level full factorial design (1331 points) used for final validation.
Key Insights and Ideas
- MLE is Unreliable with Small Samples: Standard MLE for Kriging covariance parameters can be highly unreliable and biased when the number of observations is small. The asymptotic results based on the Fisher Information Matrix do not hold well in these low-data regimes.
- Trend Choice is Crucial, Especially for Extrapolation: The choice of the deterministic trend in Kriging is not just a matter of fitting the data at hand; it critically impacts extrapolation performance. Misspecifying the trend can lead to severe prediction errors, particularly when the design is not space-filling or when predicting outside the design domain.
- Additive Models Offer Flexible Trend Modeling: Additive Models provide a powerful way to capture non-linear trends by modeling each input variable's effect independently. This flexibility is valuable when prior information about the trend is limited.
- Cross-Validation-Inspired Approach for Parameter Estimation: Estimating Kriging covariance parameters on a separate validation set (inspired by cross-validation) can lead to more accurate and robust estimates, especially when dealing with complex, non-linear trends. This approach helps to avoid overfitting the trend to the design data.
- Kriging with Additive Trend (KAT) Shows Promise: The proposed KAT method, which combines additive models for the trend with Kriging for the residuals, demonstrates good performance in the 3D application, effectively capturing both additive and potentially non-additive components of the response.
Key Results
- Parameter Estimation Variance: The empirical variances of the estimated covariance parameters (specifically, the range parameter) were found to be significantly larger than predicted by asymptotic theory when using small sample sizes (n ≤ 5). The distribution of these estimates was also observed to be more lognormal than Gaussian.
- Trend Misspecification Impact: In extrapolation scenarios, Ordinary Kriging (OK) and Universal Kriging (UK) with a misspecified linear or quadratic trend performed poorly, especially when the design points were clustered or not space-filling. UK with an affine trend showed better results when the design was regular.
- KAT Performance: In the 3D porous media example, the KAT method, particularly with a "GAM mixed" trend (linear for x1 and x3, spline for x2), showed good performance. The sequential estimation approach (Algorithm 2) led to more accurate Kriging predictors compared to a direct ML estimation on the full dataset, especially when extrapolating. The final model achieved satisfactory standardized residuals and minimized the Integrated Squared Error (ISE) on a test design.
Limitations and Future Work
- Small Sample Size Issues: The paper highlights the limitations of MLE with small sample sizes and suggests that Bayesian techniques or penalized likelihood methods might be more robust alternatives.
- Extrapolation Risks: The authors emphasize the inherent risks of extrapolation in high-dimensional spaces and the potential for trend misspecification to lead to significant errors. They suggest that in situations with no prior trend information, Ordinary Kriging might be a safer choice.
- Robustness to Design Changes: The paper acknowledges that the robustness of the proposed KAT method to changes in the experimental design has not been fully investigated and suggests this as a subject for future work.
- Further Cross-Validation-Like Methods: The authors express optimism about developing further "cross-validation-like" methods for combining additive models and Kriging to improve uncertainty quantification.
- Comparison of ML Methods: Future work could involve empirical comparisons of ML and penalized ML methods for parameter estimation, especially with classical designs.
Summary updated