Neural operators are networks that map an input physical field, such as today's velocity field or a permeability map, to an output field, such as tomorrow's velocity or the pressure. They are increasingly pretrained on many simulated equations and then fine-tuned on a new system with little data. Two habits make this fine-tuning fragile. The first is that physics enters only through an exact equation. Methods that add physics either need the closed-form equation of the new system, or they drop physics and fit the few available examples. The second is that published adaptation gains are hard to trust, because each paper uses its own splits, metrics and tuning budget. We propose to fine-tune a frozen pretrained operator through low-rank adapters, which are small trainable matrix corrections added to frozen weights. The adapters are trained against a physics penalty learned from the target data rather than written down. The penalty has the shape of a generic conservation law: the change in a quantity equals what flows across boundaries plus a source. It is applied to many unlabeled initial conditions, for which no simulated answer is needed. Everything is run under one frozen PDEBench protocol with physics-aware metrics and equal tuning budgets for every competitor.
The central bet concerns optimization as much as data efficiency. One prior study found that physics penalties make neural operators worse because they damage the loss landscape. Another found that the same kind of penalty becomes well behaved when updates are confined to a single layer. We conjecture that confining updates to a low-rank subspace of a pretrained operator plays the same stabilizing role, and that it lets a learned and imperfect physics penalty help instead of hurt. The proposal is built so that this conjecture can fail cleanly, and so that the plain alternative of carefully tuned vanilla adapters gets every chance to win.
Core Claim
In the few-shot regime (roughly 8 to 64 labeled target trajectories), a conservation-form residual fitted to the target data and enforced on unlabeled inputs will improve a low-rank-adapted pretrained operator. The improvement should appear on conservation error and long-horizon rollout error relative to a learning-rate-and-rank-tuned vanilla adapter trained on data alone. The same residual should give little benefit, or cause harm, when applied with full fine-tuning or training from scratch. That interaction between the low-rank constraint and the learned penalty is the new claim. Neither ingredient alone is new.
Concretely, let $u_t$ be the operator's predicted field at step $t$, with time step $\Delta t$. Small networks $F_\phi$ (a flux) and $q_\phi$ (a source) take as input $u$ and its finite-difference derivatives on a local stencil. The learned residual is
$$ R_\phi(u_t,u_{t+1}) = \frac{u_{t+1}-u_t}{\Delta t} + \nabla_h\cdot F_\phi(u_t,\nabla_h u_t) - q_\phi(u_t), $$
where $\nabla_h$ is a fixed discrete divergence and gradient. Because the flux enters through a discrete divergence, the integrated quantity changes only through boundary flux and the source by construction. An $L_1$ penalty on $q_\phi$ stops the source from absorbing all of the dynamics, which would make the structure vacuous. Let $\theta$ be the frozen pretrained weights and $BA$ the adapters. The adaptation objective is
$$ \min_{B,A}\; \mathcal{L}_{\text{data}}(\theta + BA;\, \mathcal{D}_{\text{lab}}) + \lambda\, \mathbb{E}_{u_0 \sim \mathcal{D}_{\text{unlab}}}\big\|R_\phi\big(\text{rollout}_{\theta+BA}(u_0)\big)\big\|^2, $$
where $\mathcal{D}_{\text{lab}}$ holds the few labeled trajectories, $\mathcal{D}_{\text{unlab}}$ holds initial conditions without solver outputs, and $\lambda$ is a tuned weight. $\phi$ is fitted first on $\mathcal{D}_{\text{lab}}$ ground truth. It is then refined for a small, fixed number of alternating rounds with the adapters.
Where This Sits
The residual design comes from the equation-agnostic physics line. PAPM builds the same convection–diffusion–source skeleton into the architecture of a proxy model trained from scratch, and reports accuracy with 5% of the training data. We reuse the skeleton as a loss on a pretrained model instead of as an architecture. PPI-NO alternates between an operator and a local pseudo-physics network fitted from data. It cuts relative $L_2$ error by 60–94% on Darcy flow and nonlinear diffusion with 5–30 samples, but trains from scratch per task and runs 60–70x slower than standard training. We keep its idea of a data-fitted local physics model, constrain that model to conservation form, and drop the per-task from-scratch training. ENO learns an energy functional and pays up to 20x training time for functional derivatives. FINDE learns and exactly preserves invariants in full neural ODEs. Both motivate learning structure from data. Neither addresses transfer, and our finite-difference residual is chosen specifically to avoid their cost.
The optimization conjecture comes from two papers that pull in opposite directions. The False Promise of Zero-Shot Super-Resolution found that adding PDE residual losses worsened operators on PDEBench tasks. This is a constraining result, and our design must not repeat it. Spectral-Refiner fine-tunes only the last spectral layer of a spatiotemporal operator under a negative Sobolev Navier–Stokes residual, which yields a convex objective. It reaches relative errors of $10^{-3}$ to $10^{-4}$ on turbulence, but it requires the exact equation and an RK4 solver. We test whether a low-rank subspace gives a similar stabilizing effect for a learned residual, where no convexity guarantee exists.
The adaptation method and the evaluation discipline come from the low-rank line. Low-Rank Adaptation of Neural Fields shows that edits to physical fields are effectively low-rank. There, learned low-rank updates beat post-hoc SVD compression by over 10 dB PSNR, but the work never touches operators. Elliptic PDE learning is provably data-efficient gives a theoretical reason to expect low-rank structure in solution operators. A Unified Study of LoRA Variants is the most important constraint on this proposal: under a unified codebase, well-tuned vanilla LoRA matches or beats elaborate variants. This is why our main competitor is tuned vanilla LoRA and not a weak default. PDEBench supplies the data and the physics-view metrics: conservation error (cRMSE), boundary error (bRMSE) and band-split Fourier error (fRMSE). Existing adaptation work does not report these. UPS fully fine-tunes a language-model backbone on PDEBench, reports only point-wise errors, and needs text prompts that name the equation. That is exactly the dependency we remove.
Research Program
We first build the backbone. We pretrain a Fourier neural operator, a network that mixes channels mode by mode in Fourier space, on a pool of PDEBench time-dependent families. We then hold out each target family in turn: 1D Burgers at unseen viscosities, 1D compressible Navier–Stokes, 2D shallow water, and 2D diffusion–reaction. The target equation is never given to the adaptation method. Diffusion–reaction is included deliberately. It is source-dominated, so it shows whether the source penalty on $q_\phi$ hurts when the conservation shape is a poor fit. A second backbone, a transformer operator, checks that the findings are not specific to one architecture.
Second, we define how adapters attach. For transformer and linear layers we use standard $BA$ adapters. For Fourier layers, which store a complex channel-mixing matrix per retained mode, we compare two placements. One places a low-rank channel correction shared across modes. The other places a per-mode correction restricted to the lowest modes, following the frequency-domain motivation of FouRA. This comparison is secondary. Its purpose is to make sure that "LoRA on an operator" is not handicapped by a poor placement before the main claim is tested.
Third, we freeze an evaluation protocol in the spirit of Standardizing Evaluation Protocols in Machine Learning, so that our own results cannot benefit from hidden degrees of freedom. The protocol fixes splits per family, fixes labeled budgets of 8, 16, 32, 64 and 512 trajectories, and fixes an unlabeled pool of 2,000 initial conditions. Every method gets the same 30-trial search over learning rate, rank, and (where applicable) $\lambda$. Five seeds are run at the selected configuration. Reported metrics are nRMSE, cRMSE, bRMSE, fRMSE in low/mid/high bands, and error at 1x, 4x and 10x the training rollout horizon. Wall-clock time and memory are measured on identical hardware. The protocol and code are released so later operator-adaptation papers can report against it.
Baselines
The naive reference point is the frozen pretrained operator applied zero-shot to the held-out family. A second naive reference is a model trained from scratch on the labeled trajectories alone. Together they show how much any adaptation helps.
The strongest competitor is vanilla LoRA trained with the data loss only, given the full 30-trial learning-rate-and-rank sweep. A Unified Study of LoRA Variants shows that this simple recipe, properly tuned, is very hard to beat. There, vanilla LoRA reached 75.51% on GSM8K with the lowest memory and runtime among variants, but that is a language result. No reported result was found on Lacuna for LoRA-adapted neural operators on PDEBench, so this baseline's numbers will be established inside our protocol. Beating it means lower cRMSE and lower 10x-horizon error at labeled budgets of 32 or fewer, outside the five-seed confidence interval, without worse nRMSE.
Full fine-tuning and last-layer fine-tuning, both data-only, get the same sweep. The last-layer variant is the equation-free analogue of Spectral-Refiner.
PPI-NO is run from its released code, BayesianAIGroup/PPI_NO, on our splits and budgets. Its reported reference is a 60–94% relative $L_2$ reduction over a plain operator on Darcy flow and nonlinear diffusion with 5–30 samples. We additionally run it on our time-dependent families and report its training time.
An oracle uses the true analytic residual of the target PDE with LoRA. It is an upper reference, not a competitor, because it violates the equation-agnostic setting.
The ablations isolate the key ingredient. In one, unlabeled inputs are used only through self-training with pseudo-labels from the adapted model, which tests whether the gain is just from extra inputs. In another, the learned residual is an unstructured local network predicting $u_{t+1}$ from a neighborhood of $u_t$, with no divergence form, which tests whether the conservation shape matters. A third drops the source sparsity penalty. The full method is crossed with full fine-tuning, last-layer tuning and training from scratch, to test the interaction claim directly.
Why not just…?
Why not just tune vanilla LoRA carefully and skip the physics? This is the real threat. The evidence from A Unified Study of LoRA Variants says that additions to LoRA usually wash out under fair tuning. Our reason to expect a difference is specific. With 8–32 trajectories, a data loss says nothing about states the labels never visit, while the learned residual is enforced on 2,000 unlabeled rollouts. FINDE shows that pure data fitting drifts badly over long rollouts. If tuned vanilla LoRA matches us on cRMSE and long-horizon error, the claim is false, and we will report it as such.
Why not just use the unlabeled inputs through self-training? Self-training also exploits unlabeled initial conditions, at almost no cost. If it matches the flux-form residual, then the physics structure is decorative. The self-training ablation and the unstructured-residual ablation together separate three possible sources of any gain: more inputs, a learned local dynamics model, and the conservation shape.
Why not just fine-tune fully or train PPI-NO from scratch? The False Promise suggests that physics penalties on unconstrained models degrade optimization. PPI-NO works from scratch but at 60–70x training cost and without pretrained knowledge. The cross of residual × update scope tests this directly. Our conjecture predicts that the residual helps most under LoRA. If full fine-tuning with the residual does just as well, the low-rank part of the claim fails and only the regularizer survives.
Why not just use the known equation when it is available? When the equation is known, Spectral-Refiner-style residuals are hard to beat, and we include that oracle for exactly this reason. The proposal targets the case where the equation is missing or incomplete. The gap between our learned residual and the oracle measures how much is lost by not knowing the equation.
Decisive Checks
Three results would settle the proposal. The first is the interaction test: on at least three of the four held-out families at labeled budgets of 32 or fewer, the flux-form residual should lower cRMSE and 10x-horizon error under LoRA. Under full fine-tuning and from-scratch training it should give no gain, or a smaller one. The second is the attribution test: the flux-form residual should beat both self-training and the unstructured residual. If it does not, the conservation shape contributes nothing beyond extra inputs. The third is the boundary test: the gains should shrink toward zero at 512 labels, and diffusion–reaction should reveal whether a misfit skeleton harms results. A harmful result there would define where the method should not be used. Training cost must also stay within a small constant factor of data-only LoRA, rather than PPI-NO's 60–70x. Otherwise the efficiency argument for adapters collapses.
References
- PAPM: A Physics-aware Proxy Model for Process Systems
- Pseudo-Physics-Informed Neural Operators: Enhancing Operator Learning from Limited Data
- Energy-consistent Neural Operators for Hamiltonian and Dissipative Partial Differential Equations
- FINDE: Neural Differential Equations for Finding and Preserving Invariant Quantities
- The False Promise of Zero-Shot Super-Resolution in Machine-Learned Operators
- Spectral-Refiner: Accurate Fine-Tuning of Spatiotemporal Fourier Neural Operator for Turbulent Flows
- Low-Rank Adaptation of Neural Fields
- Elliptic PDE learning is provably data-efficient
- A Unified Study of LoRA Variants: Taxonomy, Review, Codebase, and Empirical Evaluation
- PDEBench: An Extensive Benchmark for Scientific Machine Learning
- UPS: Efficiently Building Foundation Models for PDE Solving via Cross-Modal Adaptation
- FouRA: Fourier Low-Rank Adaptation
- Accessible, Realistic, and Fair Evaluation of Positive-Unlabeled Learning Algorithms