Machine-learned PDE solvers are now pretrained on large mixtures of simulation datasets and advertised as general-purpose. A partial differential equation (PDE) governs a field spread over space and time, such as velocity in a fluid or concentration in a reacting chemical. These pretrained models are then praised for how well they transfer to downstream equations. The test trajectories used to show this transfer usually come from the same simulation code as the pretraining data: the same recipe for drawing random initial conditions, the same grid, the same numerical solver, often the same parameter values. In language-model evaluation this would be called contamination. The model may have learned the habits of a data generator rather than the physics. No paper in this literature measures it. We propose to build, for each standard PDE benchmark, a twin test set. Each twin keeps the same equation, parameters, and difficulty but is produced by a different initial-condition recipe and an independent solver. The error gap between original and twin, corrected by a reference solver that has learned nothing, gives a direct number for how much reported generalization is generator memorization.
The question matters because the field is about to invest heavily in "foundation" neural operators on the strength of benchmark tables. If those tables mostly reward familiarity with one HDF5 generator, the community needs to know before building further on them. If the models survive the twin test, that is equally valuable. It would be the first evidence that their transfer claims hold up on data no pretraining run has seen in distribution.
Core Claim
Contamination in scientific machine learning is rarely verbatim copying. It is generator-level leakage: test instances share the initial-condition sampler, the discretization, and the numerical-dissipation signature of the solver with the training corpus. Our claim is that this leakage can be separated from real generalization with a two-by-two factorial design. One axis swaps the initial-condition generator; the other swaps the solver. Difficulty is held fixed by matching the energy spectrum (how much of the field lives at each spatial frequency) and the amplitude of the new initial conditions to the originals. A non-learned coarse numerical solver serves as a calibrator, confirming that the twin is no harder in itself. The quantity we report for each model $m$ and benchmark is
$$R_m = \frac{E_m^{\text{twin}} / E_m^{\text{orig}}}{E_{\text{cal}}^{\text{twin}} / E_{\text{cal}}^{\text{orig}}},$$
where $E^{\text{orig}}$ and $E^{\text{twin}}$ are normalized root-mean-square errors on the original and twin test sets, and the subscript "cal" denotes the coarse classical solver. $R_m \approx 1$ means the model's performance carries over. $R_m \gg 1$ means a share of its benchmark score was earned by matching the generator.
Grounding in Prior Work
The contamination literature supplies the motivation and the logic of the test. DyVal builds reasoning problems on the fly from random graphs. It found that models with strong static-benchmark scores, such as phi-1.5 and WizardMath-13B, scored near zero on freshly generated instances of the same task types. A²RBench generates abstract-reasoning tasks as verified Python programs and reports a 17.7% drop when surface symbols are remapped. Both show that changing how an instance is produced, while keeping what is asked fixed, exposes shallow fitting. Both work only on discrete symbolic tasks. We carry the same logic into continuous fields, where "remapping" means changing the initial-condition family, the solver, and the coordinate frame.
The closest existing work is Do Tabular Foundation Models Know Physics?. It audits tabular foundation models on 316 physics equations, regenerating every evaluation table from scratch, and reports a clean contamination audit. Yet all models collapse to baseline levels once asked to extrapolate. That paper both motivates and constrains ours. It shows that "resample fresh points from the same generator" can pass while out-of-generator behavior fails. It is also limited to algebraic, zero-dimensional regression, and it never changes the generator itself. Our factorial design targets exactly the leakage channel its audit cannot see. We apply it to spatio-temporal neural operators rather than tabular regressors.
On the PDE side, PDEBench supplies the benchmarks and the physics-aware error metrics. These are errors on conserved quantities, on boundaries, and in Fourier space. PDEBench itself notes that plain RMSE hides violations of physical structure. Several pretrained PDE models are evaluated against or built from such data. Among them are UPS, which adapts pretrained cross-modal models to PDE solving; OmniArch, a multi-physics foundation model; and DiffPDE, which uses a masked diffusion language model as a solver. These are our main subjects. We will document from each model's released configuration which benchmark files entered its pretraining. We will not assume overlap.
Test-time Generalization through Neural Operator Splitting trains a dictionary of single-physics operators (DISCO) and composes them at test time. It reports a normalized RMSE of 0.066 on 2D Navier-Stokes against 0.572 for the base model. It is a natural subject, because its claim is compositional generalization. That claim should be the least sensitive to generator leakage if it is real.
HyPINO uses the method of manufactured solutions as a training-data engine: pick an analytic solution, apply the operator symbolically, and read off the forcing and boundary data. It only ever uses this offline, for training. We use the same engine for a secondary, steady-state arm. There, the twin is a held-out family of analytic functions with matched spectrum. For steady problems like Poisson and Helmholtz the forcing is a native input, so manufactured solutions give exact ground truth with no solver error at all. We deliberately do not use manufactured solutions for the time-dependent arm. Doing so would add a forcing term that unforced pretrained models never saw, and this would confound leakage with an input-format shift.
Research Program
The work has three parts.
Twin generators. For each benchmark we read the original initial-condition recipe from its release. We then write a second recipe from a different function family. Examples are Gaussian random fields whose spectrum is fit to the empirical spectrum of the original test initial conditions, or piecewise-smooth fields with matched total variation. We accept a twin batch only if its spectrum, amplitude distribution, and (for shocks) gradient-magnitude distribution fall within a pre-registered tolerance of the originals.
Independent solvers. We re-simulate both the original and twin initial conditions with an independent solver built on a different scheme family, for example a spectral scheme where the original used finite volumes. We keep only cases where the two solvers agree on original initial conditions to well below the best model's error. This ensures that "solver swap" probes the model's sensitivity to solver signature rather than real solver disagreement.
Exact-overlap audit and frame tests. We run an exact and near-duplicate audit by nearest-neighbor search between test trajectories and each model's documented pretraining files. This separates verbatim leakage from generator leakage. We also apply exact symmetry transforms (rotations and reflections of isotropic problems) as a cheap frame test, motivated by Symmetry-Informed Governing Equation Discovery, which argues that frame invariance is underused in physics learning.
The factorial cells give four errors per model: original initial conditions with original solver, original with new solver, twin with original solver, and twin with new solver. Comparing them attributes any degradation to the initial-condition generator, to the solver signature, or to their interaction. We report every cell at several rollout horizons, because PDEBench documents steep error growth over long rollouts. Leakage may appear only beyond the training window.
Baselines
The naive reference point is the status quo: each model's error on the original benchmark test split, alongside a persistence predictor that repeats the last observed frame. If a model's twin error approaches persistence, its benchmark advantage was not physics. For the forward-prediction numbers themselves we will rerun all models under identical inference settings. We will not copy tables. No reported per-task error values for the foundation models on PDEBench were found on Lacuna. The only PDEBench number in the file is runtime (about 0.14 seconds for ML inference against about 60 seconds for a numerical solver on 3D fluids), which is not an accuracy baseline.
The strongest existing method for detecting contamination is the fresh-resampling audit of Do Tabular Foundation Models Know Physics?: regenerate the test set from the same generator with new random seeds and compare. We run it on every model with the same number of trajectories and solver budget as our twin sets. The proposal beats it if at least one model passes the resampling audit (ratio within 10% of one) while showing a calibrated $R_m$ above 1.5 on a twin. That would be leakage the existing audit is structurally blind to. If no such case appears across all models and benchmarks, our method adds nothing over resampling, and we will say so.
The strongest modeling baseline is a specialist FNO trained from scratch on each benchmark's original training split, with the same tuning budget we give any fine-tuned foundation model. It tells us whether a large pretrained model's leakage ratio is worse than that of a small model that has seen only one generator. For compositional generalization we use DISCO with operator splitting, whose reported 2D Navier-Stokes NRMSE of 0.066 (from Neural Operator Splitting) we first reproduce on the original data before testing it on twins.
Two ablations isolate the key ingredient, difficulty matching. In the first we build twins without spectral matching. If unmatched twins inflate the calibrator's error while matched twins do not, this confirms that matching is what makes $R_m$ interpretable. In the second we drop the calibrator normalization and report raw ratios, to show how much of an apparent gap is mere difficulty change.
Why not just…?
Why not just hold out a parameter range or keep a private test set? Held-out parameters still share the initial-condition sampler and the solver with training, so they test a different axis from the one we target. A private set drawn from the same generator is exactly the fresh-resampling audit, which the tabular-physics study passed while models still failed out of domain. We include held-out parameter splits as an extra column. If they predict $R_m$ well across models, the twin machinery is redundant, and the experiment will show it.
Why not just rotate and reflect the existing test data? This is the cheapest analogue of A²RBench's symbol remapping, and it is a real threat to our added value, because it needs no new simulation. It only applies to problems with the relevant symmetry, though, and it leaves the spectrum, the solver signature, and the initial-condition family untouched. A model can therefore be frame-robust and still generator-bound. We run the symmetry test as its own baseline. If symmetry-transform degradation ranks models the same way calibrated $R_m$ does, we will recommend the cheap test and present the twins only as a confirmation.
Why not just draw new random initial conditions and run a solver, without the factorial design? This is essentially what we do, and the contribution is not the solver run. Without spectral matching and a non-learned calibrator, any error increase is uninterpretable: the new initial conditions might simply be harder. Without the solver axis, one cannot tell whether a model learned an initial-condition family or a solver's numerical dissipation. The unmatched-twin ablation measures directly how much this naive version misleads.
Why not just check the PDE residual of model outputs on new inputs? Residual checks need no ground truth, but they are computed on discrete grids. They are noisy near shocks, and as PDEBench notes, low pointwise error and physical fidelity can diverge. We report residuals alongside error for completeness. We expect them to correlate only weakly with the twin gap, and we will test that correlation explicitly.
Proposed Experiments
The time-dependent arm uses PDEBench's 1D advection, 1D Burgers, 1D and 2D diffusion-reaction, 2D incompressible Navier-Stokes, and shallow-water tasks, plus DISCO's 2D Gray-Scott and Navier-Stokes settings. The steady arm uses HyPINO's Poisson and Helmholtz problems with manufactured solutions drawn from held-out function families. Metrics are normalized RMSE together with PDEBench's conservation, boundary, and Fourier-space errors, each at short and long horizons. Subjects are UPS, OmniArch, and DiffPDE, together with HyPINO, DISCO, and the scratch-trained FNO, wherever released weights permit.
Three checks will decide the study. First, the calibrator ratio must stay within 10% of one on every matched twin; otherwise matching has failed and that benchmark is dropped. Second, the factorial cells must show whether degradation tracks the initial-condition axis, the solver axis, or both. A model sensitive only to the solver axis would be a striking finding about learned numerical dissipation. Third, the per-model $R_m$ values must be compared against parameter-holdout and symmetry-transform degradation, to establish whether the twin protocol carries information the cheap alternatives miss. We will release the twin generators and solvers, so that future benchmarks can ship with a matched twin from the start.
References
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- A²RBench: An Automatic Paradigm for Formally Verifiable Abstract Reasoning Benchmark Generation
- Do Tabular Foundation Models Know Physics? Contamination, Units, and the Deterministic Limit
- PDEBench: An Extensive Benchmark for Scientific Machine Learning
- UPS: Efficiently Building Foundation Models for PDE Solving via Cross-Modal Adaptation
- OmniArch: Building Foundation Model for Scientific Computing
- DiffPDE: Masked Diffusion Language Models as PDE Solver
- Test-time Generalization for Physics through Neural Operator Splitting
- HyPINO: Multi-Physics Neural Operators via HyperPINNs and the Method of Manufactured Solutions
- Symmetry-Informed Governing Equation Discovery