Newton-Step Initialization of Low-Rank Adapters for Neural Operators

Scientists increasingly start from a pretrained neural operator (a network that maps an initial condition or forcing field to the full PDE solution) and adapt it to a new physical regime, such as a different viscosity or Mach number. They usually do this with low-rank adaptation, or LoRA, which freezes the pretrained weights and learns a small update written as the product of two thin matrices. The reported results in this setting are poor. LoRA leaves a "spectral error floor" that blurs fine vortices, and nonlinear adapters beat it by 16.7% to 25.4% in relative L2 error on 3D Navier-Stokes (F-Adapter). Separately, evidence suggests that the parameter shift between physical regimes really is low-dimensional (CoDA, CoLoRA). These two findings pull against each other. If the true update is low-rank, why does a low-rank adapter fail?

We propose to test one specific explanation and to build the method that follows from it. Before seeing any labeled trajectory, we write down the target PDE's residual (how badly the frozen operator's predictions violate the new equation). We take one damped Gauss-Newton step on that residual for each layer, and keep its best rank-$r$ approximation as the adapter's starting value. The adapter therefore starts as a physics-derived warm start that already moves toward the new regime, rather than at zero. This is the same move that classical physics-informed initialization makes when it seeds a network with a pseudo-inverse or a linear MMSE estimator and leaves only the residual corrections to learning. The proposal also yields a clean scientific answer. If the initialized LoRA closes the gap to nonlinear adapters, the F-Adapter error floor was largely an optimization artifact. If it does not, the floor is a genuine limit on what a low-rank update can represent, and the community should stop trying to fix it through initialization.

Core Claim

Let $W_0$ be a frozen weight matrix of the pretrained operator, $\Delta W = BA$ the rank-$r$ adapter, and $R(\Delta W)$ the vector of PDE residuals of the adapted operator's predictions. These residuals are evaluated on a batch of unlabeled input fields for the target parameter $\mu'$ and measured in the negative Sobolev norm used by Spectral-Refiner. Linearize the residual around $\Delta W = 0$ as $R \approx r_0 + J\,\mathrm{vec}(\Delta W)$, where $r_0$ is the residual of the frozen model and $J$ is the residual's Jacobian with respect to that layer's weights. The initialization is

$$\Delta W^\star = -\,\mathrm{mat}\big[(J^\top J + \lambda I)^{-1} J^\top r_0\big], \qquad B_{\text{init}}A_{\text{init}} = \mathrm{SVD}_r(\Delta W^\star),$$

where $\lambda$ is a damping constant and $\mathrm{SVD}_r$ keeps the top $r$ singular triplets. We never form $J^\top J$ explicitly. Instead we solve the inner system by conjugate gradients using Jacobian-vector products, one layer at a time.

The closest prior method is LoRA-GA. It initializes the adapter from the SVD of the plain gradient $J^\top r_0$, computed on one batch of labeled data and scaled so the model's output is unchanged at step zero. Our method differs on three points, and each one can be tested on its own. First, the signal is a label-free PDE residual rather than a data loss. Second, the step is preconditioned by curvature, so it carries a magnitude as well as a direction. Third, the step-zero output deliberately changes. Because $BA$ is a real update and not an offset subtracted from $W_0$, this also avoids the extra checkpoint storage that LoRA-GA and PiSSA need.

Relation to Prior Work

Three groups of prior work shape this proposal, and they play different roles.

Classical physics-informed initialization supplies the philosophy. GLRA deconvolution loads low-rank pseudo-inverse bases into a CNN and solves a kernel-specific matrix analytically. Learning the MMSE channel predictor seeds a network with a structured linear estimator. Both start optimization at a classical solution, but both assume the operator can be derived by hand for one fixed architecture. Our Newton step plays the role of GLRA's analytic matrix. It is computed automatically from any differentiable residual and placed inside an adapter on a large frozen operator.

Spectral initialization of LoRA in language models supplies the mechanism we extend. PiSSA initializes from the SVD of the pretrained weights. MiLoRA uses the minor singular components instead. EVA and CorDA use activation statistics, and LoRA-GA uses the gradient. All of these keep the step-zero output fixed, and none of them knows anything about a governing equation.

Scientific-ML adaptation work supplies the motivation and one constraint. F-Adapter proves a depth-amplified lower bound on the error of linear LoRA in stacked Fourier layers, and it uses only zero initialization. Because the bound concerns representation, we expect initialization to help mainly where the observed gap reflects optimization. Separating those two cases is the point of the study. CoDA and CoLoRA give evidence that regime shifts are low-rank. However, they learn the shifts with hypernetworks trained on trajectories, and they do not use frozen operator backbones with modular adapters. Spectral-Refiner shows that tuning only the last spectral layer under the $H^{-1}$ residual is convex and reaches very low error. It tunes that layer at full rank, so we borrow its loss but not its restriction to one layer.

Research Program

The backbone is a Fourier neural operator pretrained with PDEBench code (PDEBench). As a second backbone, we use PDE-Transformer, built from its released code (tum-pbs/pde-transformer). In the Fourier layers, adapters act on the channel-mixing weights of each block of frequency modes. Following F-Adapter's frequency-shell grouping, we let rank vary by shell so that the comparison with F-Adapter is like-for-like.

Initialization proceeds layer by layer from the output backwards, using residuals at the target parameter computed on unlabeled initial conditions. After that, we fine-tune on the scarce labeled trajectories with a combined loss of data error plus $H^{-1}$ residual.

We will also apply the same Newton-step initialization to the first linear map of F-Adapter's nonlinear bottleneck. This way the contribution does not depend on linear LoRA winning. Finally, we will check whether low-rank adapters restricted to Spectral-Refiner's final layer keep its convexity property, by measuring the spread of final losses across random restarts.

Baselines

The naive reference point is zero-initialized LoRA, the standard choice with random $A$ and zero $B$. It is trained with the same combined loss, the same labeled trajectories, and the same rank budget, and given three times our step budget. That allowance answers the objection that it only needs to run longer.

Full fine-tuning of the operator under the combined loss is the high-capacity reference. It is tuned over the same learning-rate grid as the other methods.

The strongest existing method is F-Adapter, run with its zero initialization and its frequency-shell allocation, at the same number of trainable parameters and the same compute. The file gives only relative figures for it: a 16.7% to 25.4% lower relative L2 error than LoRA variants on CFD-3D, and a 3.0% to 5.2% gain over nonlinear adapters without frequency allocation. I found no reported absolute L2RE on Lacuna for F-Adapter, so we will reproduce F-Adapter itself. Beating it means a lower test relative L2 error and a lower high-shell spectral error (PDEBench's Fourier-space error, fRMSE) at matched parameter count. The margin must be larger than the spread across three seeds.

For a full-rank, residual-trained reference we use Spectral-Refiner's last-layer method. Its reported Taylor-Green result is a relative error of $10^{-7}$ against $10^{-1}$ for a standard ST-FNO, and it reaches $10^{-3}$ to $10^{-4}$ on turbulent flows.

We also include PiSSA, and LoRA-GA using the gradient of the data loss on one batch.

Two ablations isolate the key ingredient. The first is residual-gradient LoRA-GA: the SVD of $J^\top r_0$ with no curvature and no magnitude. It separates the effect of physics-based residual information from the effect of the Newton step. The second, called "warm-then-project," runs $K$ steps of full-rank Adam on the residual, then truncates the result to rank $r$, with $K$ matched to the wall-clock cost of our conjugate-gradient solve.

Why not just…?

Why not just train zero-initialized LoRA longer? If F-Adapter's bound is purely a limit on representation, extra steps will not help, and neither will our initialization. If the gap comes from optimization, longer training might close it by itself. The three-times-budget baseline settles this directly. We will report the best error reached over the whole training curve, not just the error at a fixed step count.

Why not just run LoRA-GA with the PDE residual as the loss? This is the real threat, because it captures most of the idea at lower cost. Where the residual's Jacobian is well conditioned, the gradient direction and the Newton direction nearly coincide, and the curvature solve would add nothing. We expect a difference only in stiff regimes. The PINN failure-mode study shows ill-conditioning becomes severe as convection grows, so we include a coefficient sweep in 1D Advection/Burgers to look for it. If the residual-gradient ablation matches us across that sweep, we will report the cheaper method as the finding.

Why not just use F-Adapter? If our initialization only helps linear LoRA reach F-Adapter's level, the practical gain is small. That is why we also initialize F-Adapter's bottleneck with the Newton step. The claim that matters is whether a physics warm start improves every adapter family under scarce data.

Why not just fine-tune the whole operator, or only its last layer as Spectral-Refiner does? With 10 to 24 trajectories, full fine-tuning tends to overfit and lose what pretraining learned. Spectral-Refiner's one-layer method is extremely strong on periodic Navier-Stokes but tied to a Fourier basis. Our experiments include non-periodic shallow-water and magnetohydrodynamics transfers, where we can test whether its advantage holds.

Proposed Experiments

The transfers we study are the following. On 2D incompressible Navier-Stokes, we pretrain at one viscosity and adapt to viscosities an order of magnitude away, plus forced turbulence at Reynolds numbers $10^3$ to $5\times10^3$ as in Spectral-Refiner. On PDEBench CFD-3D, we pretrain at $M=1.0$ and adapt to $M=0.1$, matching F-Adapter. We also adapt to 2D shallow water with non-periodic boundaries, and to 3D magnetohydrodynamics with 24 training trajectories. Finally, 1D Burgers and Advection serve as the coefficient sweep for stiffness.

Metrics are relative L2 error, PDEBench's conservation error (cRMSE), boundary error (bRMSE), and Fourier error (fRMSE) broken down by frequency shell. We also count steps to reach a fixed error, and record initialization wall-clock time as a fraction of fine-tuning time.

Three checks are decisive. The first is the step-zero error before any labeled data is used. If the Newton initialization does not beat the frozen model, the warm start is not doing physics. The second is a rank sweep at $r \in \{4, 8, 16, 32, 64\}$ against F-Adapter at matched parameters, examined shell by shell. If initialized LoRA lifts the high-shell error floor, the floor was due to optimization; if the floor stays put at every rank, F-Adapter's bound is binding, and we will report that as a negative result. The third is the two ablations, residual-gradient LoRA-GA and warm-then-project, across the stiffness sweep. They show whether the curvature step is the ingredient that matters or whether residual-based gradients alone are enough.