Testing Simulation Error on Held-Out Operating Conditions

When engineers check a simulation against a physical prototype, they usually tune the model's uncertain parameters on every available measurement and then report how closely the tuned model matches those same measurements. Typical reports read "within 3.63%" or "excellent agreement on the first three peaks." That number answers the wrong question. A tuned model can soak up a systematic flaw, such as a missing physical effect, by distorting its parameters, so close in-sample agreement says little about whether the model will hold at a load, voltage, or speed it was not tuned on. We propose a simple test protocol that answers the right question. Hold out whole levels of an operating factor, tune the model on the rest, and run an analysis of variance on the held-out residuals (measurement minus prediction). The analysis uses repeated measurements to separate genuine, condition-dependent model error from measurement noise.

The opportunity is that the three pieces already exist separately and are combined in none of the work we found. Physical validation studies compare against all data at once. Analysis of variance (ANOVA, the classical test that splits total variation into a "between conditions" part and a "within repeats" part) is routine in agronomy and biology but is not applied to simulation residuals. Work on held-out evaluation in machine learning asks how well a model predicts on average, not whether its error is systematic. Put together, they give a test with a known false-alarm rate that can detect model-form error at new conditions and that says which factor drives that error.

Core Claim

For validation campaigns that use a few operating levels, a few repeats per level, and a model with many tuned parameters, a lack-of-fit F-test computed on factor-blocked held-out levels has more power to detect real model-form error than the in-sample test, while keeping the false-alarm rate at its nominal level. Here "factor-blocked" means whole levels are held out, not random points. The in-sample test is the one engineers would run if they ran any test at all. Calibration erases the discrepancy it is supposed to detect, and black-box tuning over many parameters makes the leftover degrees of freedom impossible to count. The advantage should grow with the number of tuned parameters $p$ and shrink as the number of operating levels $k$ grows relative to $p$. We will map where the held-out test stops paying for the data it gives up. That crossover is the formal criterion missing from the literature on when splitting is worth it.

The statistic is the standard one, applied to held-out cells. Each held-out operating condition $j$ has $r$ repeated measurements, and $e_{ji}$ is the residual of the measurement in repeat $i$. Let $\bar e_j$ be the mean residual in cell $j$, $\bar e$ the grand mean, and $k_h$ the number of held-out conditions. Then

$$F = \frac{\sum_j r\,\bar e_j^{\,2} / k_h}{\sum_{j,i} (e_{ji}-\bar e_j)^2 / \big(k_h(r-1)\big)}.$$

The numerator measures systematic miss at untouched conditions. The denominator is pure repeat-to-repeat noise. Because the held-out cells played no part in tuning, the numerator keeps its full $k_h$ degrees of freedom. With two crossed factors, the same residuals can be split further into main effects and an interaction, which shows whether the miss depends on, for example, voltage alone or on voltage and frequency together.

Prior Work and Its Roles

The motivating evidence comes from physical validation studies that report only in-sample agreement. The Chulitna bridge study compares natural frequencies from three fiber-optic strain gauges with a finite element model and reports differences of 2.40% and 3.63% (Estimation of vehicle-induced bridge dynamic responses using fiber Bragg grating strain gages). The piezoelectric pump study finds that the peak-flow frequency is shifted by one hertz relative to theory at both 100 V and 120 V, which the authors attribute to unmodeled compliance in the adhesive layer (Design and Experimental Verification of a PZT Pump with Streamlined Flow Tubes). The concrete rebar study validates wave scattering by visual agreement of peaks (Effect of Reinforcing Bar on Rayleigh Wave Propagation on Concrete Structures). The pump's repeated shift across voltages is exactly the kind of systematic, condition-linked error our test targets. None of these studies holds out data or separates such error from noise. The coarse-grained polymer study tunes 41 parameters jointly with a tree-structured Bayesian optimizer (Bayesian Optimization for High-Dimensional Coarse-Grained Model Parameterization: A Case Study on Pebax Polymer). It is the regime where an in-sample residual check is least trustworthy, because so much flexibility is fitted to the same targets.

Two papers constrain the design. The cross-validation analysis shows that K-fold cross-validation is biased toward pessimism and can lose interval coverage for slow-converging models, while the in-sample "plug-in" estimate stays valid (Is Cross-validation the Gold Standard to Estimate Out-of-sample Model Performance?). We take this seriously. That paper studies average predictive error under random splits, not testing for systematic error at new conditions, and our protocol must beat the plug-in test on its own terms or be abandoned. The small-sample paper shows that held-out estimates carry wide error bars when data are scarce (Cross-validation failure: Small sample sizes lead to large error bars). This is why we test a hypothesis with replicate-based noise estimates rather than reporting a held-out score.

One paper supplies a method we reuse. Duchi's analysis of split-conformal prediction gives coverage guarantees conditional on the one calibration split a practitioner actually holds (Sample-Conditional Coverage in Split-Conformal Prediction). We use it to attach prediction intervals for new conditions to the held-out residuals, so the protocol outputs both "is there systematic error?" and "how large an interval do I need at an unseen level?". Our contribution is not a new interval method. It is the factor-blocked split and the variance decomposition that sit underneath it.

Research Program

We will build the protocol as a small library that does four things. It takes a validation design, given as factor levels, repeats, and the model's tunable parameters. It chooses which levels to hold out, for instance interior versus edge levels, to separate interpolation from extrapolation. It calibrates on the rest with whatever optimizer the user already uses. It then reports the held-out F-test, a two-way decomposition when the design is crossed, and split-conformal intervals. Because sensor noise is often unequal across conditions, a known weakness of classical ANOVA, the library also computes a Welch-type version and a permutation version of the test that shuffles residuals within cells.

The analysis has two parts. First, for models that are linear in their parameters, we derive the power of the in-sample and held-out tests as a function of $k$, $r$, $p$, and discrepancy size. This gives a predicted crossover. The in-sample test retains $k-p$ lack-of-fit degrees of freedom but has its discrepancy partly absorbed by calibration. The held-out test has $k_h$ clean degrees of freedom but less tuning data. Second, we measure empirically how far nonlinear simulators tuned by black-box search depart from that prediction. That departure is where we expect the held-out test to matter most.

Baselines

The naive reference point is what the cited validation papers do: calibrate on all data and report in-sample percent error, as in the 2.40% and 3.63% bridge figures. We turn it into a decision rule by declaring the model adequate when in-sample error is below a fixed tolerance. It is run on exactly the same campaigns. Our protocol beats it if it flags injected model-form error that this rule passes, at a matched false-alarm rate.

The strongest competitor is the classical in-sample lack-of-fit F-test with repeats, with degrees of freedom reduced by the number of tuned parameters. It is the textbook answer, it uses all data, and the cross-validation paper's results suggest in-sample estimates are often adequate (Is Cross-validation the Gold Standard to Estimate Out-of-sample Model Performance?). It gets the same data, the same optimizer, and the same tuning budget. For black-box calibration we give it the most favorable degrees-of-freedom count we can justify. We found no reported result on Lacuna for the power or false-alarm rate of any discrepancy test on these benchmarks, so both methods will be measured from scratch. Beating it means higher power at nominal 5% false alarms in the many-parameter regime, together with no meaningful power loss when $p$ is small. A loss in the few-parameter regime is the expected outcome and defines the crossover.

A third baseline is random K-fold cross-validation of residual error followed by the same F-test on pooled out-of-fold residuals. It isolates whether the gain comes from holding out data at all or from holding out whole factor levels. The key ablation compares factor-blocked and random holdouts at an identical held-out fraction, plus parametric and permutation versions of the test under unequal noise. For prediction intervals, the reference is split-conformal intervals without factor blocking, following Sample-Conditional Coverage in Split-Conformal Prediction, and success means holding nominal coverage at held-out levels where the random-split intervals under-cover.

Why not just…?

Why not just run the classical in-sample lack-of-fit test? For a model with two or three parameters that enter linearly, this may well be as good or better, and we expect our own power analysis to show it. The threat is real only in the opposite regime. With dozens of parameters tuned by black-box search, as in the 41-parameter polymer model, calibration fits the discrepancy, and no honest degrees-of-freedom count exists. The power-versus-$p$ experiment settles which regime a given campaign is in. If the in-sample test matches ours across the range, the proposal reduces to a recommendation not to split, which would still be a useful result.

Why not just report held-out error, or conformal intervals, without the variance decomposition? A held-out error of 5% could be pure sensor noise or a consistent bias. Its error bars at small sample sizes are wide (Cross-validation failure: Small sample sizes lead to large error bars), so it cannot tell the two apart. Conformal intervals can be valid while being wide precisely because of an unmodeled effect, and they do not say which factor causes it. The ANOVA's within-repeat noise estimate and its interaction term supply exactly that missing information. Our experiments check that the decomposition attributes injected error to the correct factor.

Why not just add a discrepancy term to a Bayesian calibration? Fitting a flexible model-error function alongside the physical parameters is a principled alternative. With few operating levels, however, the discrepancy function and the physical parameters trade off against each other, the same equifinality the polymer study runs into. Its conclusions then depend on the prior. No reported result found on Lacuna for this approach on our benchmarks. We will include a Gaussian-process discrepancy model as an exploratory comparison on the toy testbed, judged on the same false-alarm and power criteria.

Proposed Experiments

Testing requires knowing the truth, so the main experiments use simulated "physical" campaigns with controlled injected flaws. The first testbed reuses the low-dimensional damped oscillator and wave problems from the suite in PIED: Physics-Informed Experimental Design for Inverse Problems. "Reality" includes an extra compliance or damping term that shifts resonance with drive level, mimicking the pump's one-hertz shift. The model omits that term and is calibrated over 2 to 20 parameters. We sweep $k$ from 4 to 12 levels, $r$ from 2 to 5 repeats, the noise level, and the discrepancy size. The second testbed uses the Kolmogorov-flow setting from Factorized Fourier Neural Operators. There the pseudo-spectral solver at a fine resolution plays reality, with noisy point sensors and viscosity and forcing as crossed factors. The model is a trained neural operator or a coarse solver with tunable viscosity. This adds high-dimensional output fields and a calibration step costly enough that a black-box optimizer is the natural choice. As a real-data check in the regime where in-sample estimation is known to be safe, we use the puma32H robot-arm data from Is Cross-validation the Gold Standard to Estimate Out-of-sample Model Performance?. We bin one input into factor levels and ask whether the held-out test spuriously flags well-specified models.

The metrics are the empirical false-alarm rate under no injected flaw, measured over 2,000 replicated campaigns per setting. We also measure power against discrepancy size, correct attribution of the injected flaw to its factor in the two-way decomposition, error in the estimated extrapolation miss compared with the known truth, and conformal coverage at held-out levels. The decisive checks are three. First, the held-out test must keep false alarms near 5%, including under unequal noise when the permutation version is used. Second, it must beat the corrected in-sample test in power once $p$ is a substantial fraction of $k$, with the observed crossover matching the linear-model prediction to within the sampling error. Third, factor-blocked splitting must outperform random splitting at equal held-out fraction. If the second check fails across the board, the honest conclusion is that in-sample testing suffices and data should not be split. We would report that as the finding.