Regime-Stratified, Experiment-Anchored Evaluation of Airfoil Flow Surrogates

Machine-learning surrogates are neural networks trained to imitate slow physics solvers, and they now return airfoil flow fields in milliseconds instead of the roughly 25 minutes a full simulation takes. They are ranked by one averaged error against the solver's output. That single number hides two separate problems. The first is that the surrogate's error is not spread evenly. It may be small on gentle, attached flows and large where the flow peels off the wing (separation, near stall), and an average over a dataset dominated by easy cases washes that out. The second is that the solver used as "ground truth" is itself only an approximation of real air, and its own error also depends on the flow regime. Where the solver disagrees with a wind tunnel by more than the gap between two competing surrogates, ranking those surrogates in that regime says nothing about physical accuracy.

We propose to separate these two errors explicitly. Every surrogate's error is reported per physical flow regime. A small wind-tunnel campaign then measures how far the solver itself sits from reality in each regime, and this anchors each regime's error. The claim we test is concrete: on the AirfRANS benchmark, choosing models by the aggregate score picks different models than choosing by worst-regime error. In the separated-flow regime, surrogate rankings computed against the solver will not predict rankings computed against measured lift and drag.

Core Claim

Two lines of work motivate this, and neither covers the other. The first shows that aggregate scores hide failures concentrated in particular regimes. Traffic forecasting models with an overall error of about 3 mph reach 11 mph during free-flow/congestion transitions, and their 90% interval coverage falls to 55% (Do Time Series Foundation Model Benchmarks Hide Regime-Dependent Failures?). In novel-view synthesis, a trivial copy baseline looks competitive on average but is 125% better on simple objects and 45% worse on complex ones (Aggregate Metrics Hide Shortcut Regimes). A further result is that choosing models by mean validation accuracy actively favours shortcut-reliant models, which its authors call the "validation trap" (Efficient Bias Mitigation Without Privileged Information).

The second line is the surrogate benchmarks, which already suspect that aggregates mislead but stop short of stratifying. AirfRANS shows that surrogates with low field error predict lift well (Graph U-Net reaches a Spearman rank correlation of 0.967 for lift) yet give "poor or even negative" correlation for drag (AirfRANS). PDEBench adds boundary, conservation and frequency-space error terms, but averages them over whole test sets (PDEBench). LIPS scores physics compliance and out-of-distribution behaviour (LIPS). All of these, along with the large search effort in SENPAI, measure fidelity only against solver output.

The engineering validation literature shows why that last point matters. Solver predictions and physical tests disagree by amounts that depend on details the model omits. In a bellows torsion test, nominal finite-element analysis was 3.3% off the measured critical torque, and accounting for a 0.02 mm wall thinning brought it to 0.9% (Torsional Buckling of U-Shaped Bellows). Bridge modal frequencies differed from the model by 2.40–3.63% (FBG bridge study). What is new here is joining the two lines. We stratify surrogate evaluation by physical regime, and we use a per-regime physical discrepancy floor to decide where surrogate differences are meaningful at all.

Research Program

The first step defines strata without circularity. The aggregate-bias literature warns that stratifying by a quantity that shares the evaluation metric's failure modes is circular. We therefore stratify only on simulation inputs (angle of attack, Reynolds number, camber, thickness) and on one solver-derived flow descriptor that no surrogate touches: whether and where the reference RANS solution shows negative wall shear stress, meaning the flow has separated, together with the peak adverse pressure gradient on the suction side. RANS is the Reynolds-averaged Navier–Stokes solver that AirfRANS uses. These choices yield attached, mild-adverse-gradient and separated strata across the AirfRANS training range.

The second step is a stratified audit. We train a pool of surrogates on the AirfRANS full-data split under identical budgets: MLP, PointNet, GraphSAGE, Graph U-Net and Transolver, with several seeds and hyperparameter settings each. For every model and stratum we report surface pressure error, wall-shear error, and lift and drag error and rank correlation. Selection under the aggregate criterion is then compared with selection under the worst-stratum criterion.

The third step anchors the strata in experiment. We would test four to six NACA sections that span the camber and thickness strata in a wind tunnel at a chord Reynolds number near $2\times10^6$, the low end of the AirfRANS range. Surface pressure taps give the pressure coefficient and integrated lift, and a wake rake gives drag, over a sweep of angles that crosses the onset of separation. For each stratum $s$ we compute

$$r_s = \frac{e_s(\text{surrogate}, \text{solver})}{e_s(\text{solver}, \text{experiment})},$$

where $e_s$ is the mean absolute lift or drag coefficient error within stratum $s$. When $r_s \ll 1$, the surrogate is already more faithful to the solver than the solver is to reality, so further leaderboard gains in that stratum carry little physical meaning. When $r_s \gtrsim 1$, the surrogate is the bottleneck. The file indexes no wind-tunnel dataset for these sections, so this measurement campaign is a genuine cost of the project, and the program is designed so that the first two steps stand alone if it slips.

Baselines

The naive reference is the current practice: rank surrogates by aggregate surface MSE and volume MSE on the AirfRANS test split, with no stratification. Everything below is compared against the model this procedure would select.

The strongest existing method is the Transolver architecture as reported on AirfRANS, with surface MSE of 0.0142, and the SENPAI-discovered training recipe, which reports surface MSE of 0.00130 (SENPAI). We would reproduce Transolver under our shared budget, and include the SENPAI recipe if its configuration is available. Otherwise we report SENPAI's number as a reference only, without claiming a matched comparison.

The strongest existing physics-aware evaluation is the AirfRANS drag and lift rank correlation together with its Reynolds and angle-of-attack extrapolation splits. These are the natural competing way to expose hard regimes. Our strata must reveal failures those splits miss for the proposal to add anything.

No per-regime surrogate results and no experiment-referenced surrogate results were found on Lacuna for AirfRANS. The proposal's numbers are therefore new measurements rather than improvements over a reported figure. "Beating" a baseline here means one of two things. Either the worst-stratum criterion selects a model whose separated-stratum drag error is lower than that of the aggregate-selected model while aggregate surface MSE stays within seed noise, or experiment-referenced rankings disagree with solver-referenced rankings in the separated stratum.

The key ablation removes the experimental anchor. We run the same stratified selection but weight strata uniformly instead of by $r_s$. This isolates whether the physical measurements change any decision, or whether stratification alone does all the work.

Why not just…?

Why not just use AirfRANS's existing angle-of-attack and Reynolds extrapolation splits? Those splits move test cases outside the training input range. Separation, however, also occurs inside the central angle range for thick or highly cambered sections, so in-distribution failures stay averaged away. We include both splits as stratifiers and report whether our separation-based strata identify model rank changes that the extrapolation splits do not. If they do not, the extrapolation splits suffice, and we would say so.

Why not just report boundary-error and drag-correlation metrics, as PDEBench and AirfRANS already do? These metrics are better targets but are still averages over the whole test set. The traffic result, where a 3 mph mean hid an 11 mph transition error, shows the same washout can occur for any averaged quantity. The direct test is to compute them per stratum and check whether their aggregate value predicts the worst-stratum value across the model pool.

Why not just correct the solver against experiments and train surrogates on the corrected data? This is a real threat to the experimental half of the proposal. If the measured solver–experiment discrepancy turns out small and uniform across strata, the anchor adds nothing, and the work reduces to the purely computational stratified audit. Step three exists to measure exactly this. A flat $r_s$ denominator across strata would be reported as a negative result for experiment anchoring.

Why not just train with worst-group reweighting, in the style of TAB? Reweighting changes training. The question here is how models should be evaluated and selected, and a reweighted model still needs a trustworthy way to judge it. We include stratum-reweighted training as an additional pool member, so the audit can show whether it closes the separated-stratum gap.

Proposed Experiments

The first decisive check measures rank agreement across the model pool on AirfRANS. We compute Kendall's tau between models' aggregate surface MSE and their separated-stratum drag error. A tau well below the seed-to-seed agreement, together with a different selected model, would show the validation trap in surrogate selection.

The second check is whether aggregate gains concentrate in easy strata. For each pair of successive improvements in the pool, such as Graph U-Net to Transolver, we attribute the reduction in aggregate error to strata. The hypothesis is that most of the gain comes from the attached stratum.

The third check, which requires the wind-tunnel data, tests whether solver-referenced and experiment-referenced lift and drag errors rank the top models the same way, separately for the attached and separated strata. The central prediction is agreement in attached flow and disagreement in separated flow, where $r_s$ falls below one.

Failing any of these predictions is informative. Agreement everywhere would mean the AirfRANS leaderboard is physically trustworthy within its range, which is itself a result no current benchmark can claim.