Optimal Calibration for Computer Model Prediction with Finite Samples

Xiaowu Dai, Peter Chien

Abstract

This paper addresses the challenge of computer model calibration within a non-asymptotic frequentist framework. Traditional calibration methods often rely on asymptotic assumptions—assuming an infinite amount of data—which rarely align with the reality of engineering and physical sciences where data is scarce and models are inherently inadequate. We propose an "optimal calibration" approach that minimizes the predictive mean squared error (PMSE) for finite samples. By modeling the discrepancy between the computer simulation and the physical system within a Reproducing Kernel Hilbert Space (RKHS), we demonstrate that minimizing the RKHS norm, rather than the standard $L_2$ norm, provides superior predictive performance. Our framework includes an iterative algorithm for practical implementation and establishes a theoretical bridge to existing Bayesian calibration techniques.

Introduction

In modern scientific inquiry, computer models serve as essential proxies for complex physical systems, ranging from climate simulations to material stress tests. However, two fundamental issues persist: model inadequacy and finite observational data. As George Box famously noted, "All models are wrong." Even the most sophisticated simulations fail to capture every nuance of reality. Furthermore, physical experiments are often prohibitively expensive or time-consuming, leaving researchers with a limited number of observations to calibrate their models.

Existing statistical calibration literature largely focuses on asymptotic properties. While these theories are mathematically robust, they often fail to provide reliable guarantees when sample sizes are small. This work shifts the focus toward a non-asymptotic approach, specifically designed to optimize prediction accuracy—measured by Predictive Mean Squared Error (PMSE)—under the constraints of finite data.

The Framework: Prediction-Oriented Calibration

The Discrepancy Function and RKHS

To account for model inadequacy, we define the relationship between the physical system $y(x)$ and the computer model $f(x, \theta)$ as: $$y(x) = f(x, \theta) + \delta(x) + \epsilon$$ where $\delta(x)$ represents the model discrepancy and $\epsilon$ is the observation noise.

We propose modeling the discrepancy function $\delta(x)$ within a Reproducing Kernel Hilbert Space (RKHS). The choice of the RKHS is pivotal; it allows us to leverage the "smoothness" of the error function. Our primary theoretical contribution is the "Opt-Pred" estimator. We define the optimal calibration parameter $\theta^{opt-pred}$ as the value that minimizes the estimation risk of the discrepancy.

RKHS Norm vs. $L_2$ Norm

A significant insight of this research is that the choice of norm matters immensely for prediction. Traditional Least Squares (LS) calibration typically seeks to minimize the $L_2$ norm (the distance between the model and the data). However, we prove that for optimal prediction, one should minimize the discrepancy under the RKHS norm. While $L_2$ calibration focuses on fitting the existing data points, the RKHS norm accounts for the complexity and regularity of the discrepancy function, leading to more robust generalizations on unseen data.

Methodology and Algorithm

Iterative Optimization

Finding the optimal $\theta$ and the discrepancy function simultaneously is a complex optimization problem. We provide an iterative algorithm that alternates between two steps:

  1. Discrepancy Estimation: Given a fixed $\theta$, estimate $\delta(x)$ using kernel-based methods.
  2. Parameter Update: Refine $\theta$ by minimizing the RKHS-based objective function.

This process typically begins with a simple least-squares estimate and refines the parameters through a one-step or fully iterative update procedure. We utilize Generalized Cross-Validation (GCV) to select hyperparameters, ensuring the model remains data-driven.

The Bayesian Connection

Our frequentist approach is not isolated from the popular Bayesian frameworks. We demonstrate that our optimal calibration is the limiting case of the Kennedy-O'Hagan (KO) Bayesian calibration model as the prior variance of the parameters goes to infinity. This provides a frequentist justification for certain Bayesian practices while offering a more computationally direct path for prediction.

Experimental Results

Synthetic Benchmarks

In 1D and 2D synthetic simulations, the proposed method was tested against varying noise levels. In a 1D example with low noise, the "Opt. Cal." method achieved a PMSE of 0.0922, significantly outperforming standard Least Squares calibration (0.1152) and pure Nonparametric regression (0.1701). This demonstrates that using the computer model as a "guide" for the nonparametric predictor allows for higher accuracy with fewer data points.

Real-World Application: Sodium Ion Channel Data

We applied the method to a cardiac cell membrane dataset involving voltage clamp experiments. The goal was to model normalized current over time—a critical task for understanding cardiac electrophysiology.

Method PMSE
Proposed Optimal Calibration $1.6323 \times 10^{-4}$
Least Squares Calibration $3.05 \times 10^{-4}$
Kennedy-O'Hagan (Bayesian) $4.5 \times 10^{-3}$

The proposed method achieved an order of magnitude improvement over the standard Bayesian approach and nearly doubled the accuracy of the standard LS calibration. Furthermore, the method exhibited lower standard errors, indicating higher stability in the face of noisy, real-world data.

Conclusion and Future Directions

This work provides a rigorous foundation for calibration when data is limited and models are imperfect. By prioritizing prediction over simple data fitting and utilizing the RKHS framework, we offer a method that is both theoretically sound and practically superior for finite samples.

Future research will explore the integration of derivative data—common in engineering where rates of change are measured alongside absolute values—into the RKHS framework. Additionally, we aim to address the computational challenges of kernel selection in high-dimensional input spaces to extend the reach of optimal calibration to even more complex physical systems.

Summary updated

Abstract

This paper considers the computer model prediction in a non-asymptotic frequentist framework. Two main issues arise for the prediction: (1) many computer models are inadequate for physical systems and (2) only finite samples of physical observations are available for estimating model discrepancy and calibrating unknown parameters in computer models. In this work, we propose the optimal calibration and give exact statistical guarantees in the sense that the predictive mean squared error is minimized with the optimal calibration for any finite samples. We give an equivalent formulation of the optimal calibration which leads naturally to an iterative algorithm. The connection is built between the optimal calibration and the Bayesian calibration in Kennedy and O'Hagan [J. R. Stat. Soc. Ser. B. Stat. Methodol. 63 (2001) 425-464]. Numerical simulations and a real data example show that the proposed calibration outperforms the existing ones in terms of the prediction.
BibTeX
@misc{dai2018optimal,
  author = {Xiaowu Dai and Peter Chien},
  title = {{Optimal Calibration for Computer Model Prediction with Finite Samples}},
  year = {2018},
  eprint = {1802.00021},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1802.00021},
}
Figure