Enforcing Known Conservation Laws on Frozen PDE Foundation Models at Test Time

Large pretrained models of physical simulation are now evaluated "zero-shot": they are asked to predict a fluid, plasma, or reaction system they never saw in training, with no gradient updates allowed. They often score well on average error, yet nothing stops their predictions from creating mass from nothing, letting a magnetic field develop divergence, or drifting in energy over a long rollout. In most real settings we know some of the physics of the new system, such as a conservation law or an incompressibility condition, even when we do not know the full governing equation. This proposal asks whether that partial knowledge, applied only at test time as a cheap correction to the frozen model's outputs, can make zero-shot prediction both more accurate and more physically trustworthy.

The idea is simple. After each step of an autoregressive rollout (the model predicts the next frame and then feeds that prediction back in), we project the predicted field onto the set of states that satisfy the known constraints. We choose the projected state closest to the model's guess, so the model's learned dynamics are kept and only the constraint-violating part is removed. The model's weights never change, so the evaluation stays strictly zero-shot. The size of the violation before projection then serves a second purpose: a label-free warning signal that a given zero-shot prediction is about to fail.

Core Claim

For frozen PDE foundation models evaluated on out-of-distribution systems, enforcing only a small set of known linear or low-dimensional constraints at every rollout step should do three things. It should reduce long-horizon error. It should shrink the worst-case tail of errors more than it shrinks the mean. And the pre-projection constraint residual should rank-correlate with true zero-shot error well enough to flag failures without ground truth. A PDE, or partial differential equation, is the usual law of how a continuous field changes over space and time. The constraints we use are fragments of such laws, for example divergence-free velocity or magnetic fields, conservation of total mass, and conservation of the integral invariants of KdV, the Korteweg–de Vries soliton equation. Because only fragments are assumed, a classical numerical solver cannot simply be run instead. That is the regime where this correction is not redundant.

Relation to Prior Work

Several papers motivate the gap. OmniArch pretrains a multi-physics Fourier-plus-Transformer model and evaluates it zero-shot on Orszag–Tang vortex, shocks and Kelvin–Helmholtz instability. Its physics alignment is a contrastive match to text descriptions of equations, not a residual on the predicted field, and it reports degradation at discontinuities. Physics-informed Temporal Alignment shows that autoregressive PDE models learn to copy the previous frame and drift. It adds discovered-PDE losses during training, cutting out-of-domain Burgers error by 37.8%. Our correction acts only at inference and needs no equation discovery. PDEBench motivates the evaluation side. It introduced conservation error (cRMSE) and Fourier-spectrum error (fRMSE) precisely because grid error hides physical violations. Most zero-shot reports, including those on Air Quality Arena, still report only statistical error.

A second group shares the test-time philosophy but differs in what it assumes. Test-time Generalization through Neural Operator Splitting composes single-physics operators by beam search, reaching 0.066 NRMSE on 2D Navier–Stokes versus 0.572 for the unsplit base model. This requires a purpose-built operator dictionary rather than an arbitrary monolithic model. Origo builds splitting into the architecture itself, reducing error on unseen KdV and Allen–Cahn by 50–60% over MOE-POT. SCaSML defect correction refines surrogates at inference by solving a PDE for their error with Monte Carlo. It needs the full semilinear parabolic equation. Learning a Neural Solver for Parametric PDEs minimizes the full physics loss in 2–5 learned steps. It requires a solver trained per equation family. Our approach differs from all four in the same way: it assumes only constraint fragments and touches neither the architecture nor the weights.

Two papers constrain the design. The Grad-Shafranov operator study found that physics-only objectives fall into wrong solution branches, while data-only models have fat tails, with OOD p99 error at 53.7%, cut to 44.1% by physics anchoring. This is why we anchor to the model's prediction instead of minimizing residuals freely. PINO supplies a method we reuse: exact spectral differentiation for evaluating residuals on periodic grids. Its instance-wise fine-tuning with the full PDE loss also becomes one of our baselines. Physics-informed GNNs for dynamical systems found that explicit constraints act as a "cheat code" for zero-shot scaling of particle systems. That result is evidence that the effect we hope to see exists, though only for ODE particle systems rather than PDE fields.

Research Program

At each rollout step $t$, let $\hat u_t$ be the frozen model's predicted field. Let $C(v)=0$ collect the known constraints on a candidate field $v$, and let $W$ be a positive weighting matrix. The corrected state is

$$\tilde u_t = \arg\min_{v} \; \|v - \hat u_t\|_W^2 \quad \text{subject to} \quad C(v) = 0,$$

and $\tilde u_t$, not $\hat u_t$, is fed back to the model. For linear constraints the projection has a closed form. Divergence-free fields are handled by a Helmholtz–Leray projection computed by fast Fourier transform. Total mass is handled by a uniform shift, or by a shift weighted toward regions of high predicted variance. For a few nonlinear invariants, such as KdV energy, we take two or three Gauss–Newton steps on a scalar Lagrange multiplier. The cost is a handful of transforms per step, far below a forward pass of the model.

We will study the choice of $W$, the plain Euclidean norm versus a spectral weighting that penalizes changes to high wavenumbers. The aim is to avoid smoothing away the fine structure that OmniArch already struggles to produce. We also record the pre-projection residual $r_t = \|C(\hat u_t)\|$ along each trajectory. We test whether its running maximum predicts the final normalized error across trajectories, measured by Spearman correlation and by the area under the curve for flagging the worst 10% of rollouts.

The frozen models will be released checkpoints of the public foundation models where these exist. As a controlled fallback, we will pretrain a Fourier-Transformer model on a subset of PDEBench with entire equation families held out. That fallback guarantees the test systems are truly unseen.

Baselines

The naive reference is the frozen model's raw zero-shot rollout with no correction. This is the protocol of current zero-shot reports. Any gain we claim is measured against it on identical initial conditions and horizons.

The strongest existing competitor on the composite-physics question is the operator-splitting method of Test-time Generalization through Neural Operator Splitting. It reports 0.066 NRMSE on 2D Navier–Stokes, against 0.572 for its base model. We cannot apply splitting to a monolithic model. So the fair comparison runs our projection on the splitting paper's own unsplit base model, using its Navier–Stokes split. We also run projection on top of the split model, to test whether the two gains add. We count as beating it if projection on the unsplit model closes at least half the gap between 0.572 and 0.066, or if projection on the split model lowers 0.066 further. Both are measured under their evaluation script, with matched wall-clock budgets reported. On unseen equations, Origo is the reference: we report whether projection on a generic model reaches its 50–60% reduction relative to MOE-POT on KdV.

A gradient-based test-time competitor is PINO-style instance fine-tuning with the full PDE loss (PINO). It breaks the zero-shot rule and assumes full physics, so it is an upper-reference rather than a peer. It is given the same wall-clock budget as our method and, separately, ten times that budget.

The key ablation removes the anchor and minimizes $\|C(v)\|^2$ alone from $\hat u_t$. A second ablation projects only once at the end of the rollout instead of at every step. The first isolates the role of staying close to the model, which the Grad-Shafranov branch-trapping result predicts is essential. The second isolates whether feeding corrected states back is what prevents drift.

Why not just…?

Why not just fine-tune the foundation model on a few frames of the target system? It needs target data and gradient updates, so it leaves the zero-shot regime. The splitting paper reports that gradient-based adaptation such as GEPS diverged in long rollouts and was 2.7× slower than discrete test-time selection. We will nevertheless run few-frame fine-tuning with a matched budget, both alone and combined with projection. If fine-tuning alone matches projection's tail reduction, our claim weakens to "projection is the cheaper option."

Why not just run a numerical solver warm-started from the model's prediction? This is the most serious threat whenever the full PDE is known. A converged solver makes the model's role trivial, and OmniArch's 155× speedup over finite differences shows what is lost in time. We address it directly with a "partial-knowledge" split. On systems where we supply the full equation, a warm-started solver is a baseline we expect to win on accuracy, and we will say so. The claimed contribution is restricted to systems where only constraints are given, such as unknown closures or unknown reaction terms. There no solver can be run, and we verify this by hiding the closure term from all methods.

Why not just add physics losses during pretraining, as PITA and PINO do? Training-time regularization cannot know which constraints the unseen target system obeys, and it costs 20–40% overhead per the PITA report. It is also complementary rather than competing. We test projection on a PITA-style regularized model as well as a plain one. If the gain vanishes on the regularized model, training-time physics already captures what projection adds.

Why not just scale the model? Scale alone has not delivered physical consistency. PROST found text models near chance (23–31% against 25%) on physical reasoning, and PITA documents frame-copying shortcuts in large autoregressive models. We test this by applying projection to two model sizes of the same family and checking whether the gain shrinks with scale.

Proposed Experiments

The test suite uses held-out families. Incompressible 2D Navier–Stokes and diffusion-reaction come from PDEBench, and KdV and Allen–Cahn follow the Origo protocol. Orszag–Tang and Kelvin–Helmholtz follow the OmniArch protocol, where divergence-free magnetic fields and mass conservation are the known constraints. Allen–Cahn has no simple conserved quantity, so it serves as a negative control where we expect no gain. For metrics we report normalized RMSE at short and long horizons, PDEBench's cRMSE and fRMSE, and the p90 and p99 of per-trajectory error. The tail quantiles follow the Grad-Shafranov study's emphasis on fat tails.

The decisive checks are four. First, long-horizon nRMSE and p99 error both fall relative to the raw rollout on systems with genuine constraints, with no gain on the Allen–Cahn control. Second, fRMSE does not worsen, showing the projection does not achieve conservation by smoothing. Third, the unanchored ablation is worse than the anchored one, confirming the branch-trapping concern. Fourth, the pre-projection residual flags the worst decile of raw rollouts well above chance. If the first check fails, the proposal is wrong. If the first passes but the second fails, the correction trades one physical error for another and needs the spectral weighting to be useful. If only the fourth succeeds, the result is still worth reporting: a cheap, label-free way to tell when a zero-shot physical prediction should not be trusted.