Curvature-Aware Hypergradients for PINN Loss Weights

A physics-informed neural network (PINN) is a network trained to solve a differential equation. Its loss adds up several terms: how badly the network violates the equation at interior points, and how badly it misses the boundary and initial values. Training often fails because one term's gradient swamps or opposes the others. Most fixes reweight the terms with hand-built rules. These include matching gradient sizes, matching gradient variances, and tracking how fast each loss falls. A principled alternative is to ask directly how changing a weight would change the final solution. That is a bilevel question: the outer level picks the weights, and the inner level trains the network under those weights. Answering it normally requires multiplying by the inverse of the network's Hessian, the matrix of second derivatives of the loss. That is the "Hessian inversion bottleneck," and it is especially painful for PINNs, because their residuals already contain second derivatives with respect to the inputs.

This note argues that PINNs have structure that removes the bottleneck instead of approximating it away. The PINN loss is a weighted sum of squared residuals over a batch of points. Its Gauss-Newton curvature, the standard least-squares approximation to the Hessian, can therefore be inverted in "sample space": the solve involves a batch-sized matrix of residual-Jacobian inner products, not a parameter-sized matrix. The solve needs only forward- and reverse-mode Jacobian-vector products of the residual map, and never the third derivatives that exact Hessian-vector products through a PDE operator would demand. The resulting weight update is a curvature-preconditioned measure of whether upweighting a term helps or hurts every other term. The central empirical question is whether that curvature actually matters. Existing first-order bilevel loss balancing throws it away and works on image tasks. We predict this truncation breaks on stiff PDEs, and the proposal is built to test that prediction directly.

Core Claim

Let the inner loss be $f(\lambda,\theta)=\sum_i \lambda_i L_i(\theta)$, where $L_i$ is the mean squared residual of term $i$ (PDE, boundary, initial, or data) and $\theta$ are network weights. Let the outer objective be $F(\theta)=\sum_i \log L_i^{\text{val}}(\theta)$, the log of each term evaluated on fresh, held-out points. The log makes $F$ unchanged when any term is rescaled, so the outer objective needs no weights of its own and is not circular. It rewards proportional progress on every term. It also strongly penalizes letting any term grow, which directly targets the "vanishing constraint" failure of adaptive weights noted by Niu et al..

By the implicit function theorem, the hypergradient (the gradient of $F$ with respect to the weights, accounting for how training responds to them) is

$$\frac{dF}{d\lambda_j} = -\,\nabla_\theta L_j^{\top}\, G^{-1}\, \nabla_\theta F, \qquad G = J^{\top} W J + \mu I,$$

where $J$ stacks the Jacobians of all batch residuals with respect to $\theta$, $W$ is diagonal with entry $\lambda_i/N_i$ for each residual of term $i$ ($N_i$ points), and $\mu$ is a damping constant. The Woodbury identity turns the parameter-space inverse into a batch-space solve:

$$G^{-1} v = \tfrac{1}{\mu}\left(v - J^{\top}\left(\mu W^{-1} + J J^{\top}\right)^{-1} J v\right).$$

Here $JJ^{\top}$ is the empirical tangent kernel on the $N$ batch points. The formula shows what the hand-built rules miss. Wang-style annealing compares gradient sizes. A truncated first-order bilevel rule with $G=I$ compares raw gradient directions. The curvature-aware rule compares directions in the metric the network actually moves in. That metric differs from the raw one most sharply on stiff PDEs, where a few residual directions dominate $JJ^{\top}$.

The claim has two parts. First, on stiff PDE benchmarks, the identity-truncated hypergradient points in a meaningfully different direction from the Gauss-Newton one, while on standard multi-task vision the two agree. Second, the Gauss-Newton weighting yields lower solution error at a wall-clock cost comparable to gradient-magnitude annealing.

Relation to Prior Work

Three works already join bilevel optimization to loss balancing, and each leaves this question open.

Meta-learning PINN loss functions learns loss parameters offline across a family of PDEs by unrolling a short inner trajectory. We instead adapt weights online for a single PDE and replace unrolling with an implicit, curvature-aware solve.

LDC-MTL is the closest method. It casts multi-task loss balancing as a bilevel problem, minimizes pairwise loss gaps, and drops second-order interaction terms to reach constant cost. It was evaluated only on CelebA, Cityscapes, NYU-v2 and QM9. Our proposal keeps exactly the curvature it discards, and it tests whether discarding it is safe when residuals involve differential operators.

DATS weights or samples whole PDE tasks in meta-learning. It does not address imbalance among terms within one PDE.

The heuristic weighting literature motivates the gap. Wang et al. diagnosed boundary gradients drowning out the residual and fixed it by matching gradient magnitudes. Maddu et al. matched gradient variances instead. ReLoBRaLo uses only loss ratios to stay cheap.

Two works constrain the design. Luo et al. (CAML) show that residual and boundary gradients can point in opposite directions even when loss values look balanced. That is why a pure loss-gap outer objective is unlikely to suffice. Meng et al. show that input-coordinate Jacobians can collapse or explode in deep PINNs, which no scalar weighting can fix. We therefore restrict to depths where that pathology is absent and report it as a limit.

On the bilevel side, the inversion-free penalty method of Mehra and Hamm, the value-function method BOME, and the penalty-Lagrangian methods F2SA/F3SA all avoid the Hessian entirely. FSLA tracks the inverse-Hessian product with an auxiliary state. All of these were tested on classification, hyper-cleaning, or continual learning, and most guarantees assume a strongly convex inner problem, which PINNs violate. Liao et al. approximate the inner Hessian with Kronecker-factored blocks. That is the structured-curvature competitor most similar to ours. It differs in that it factorizes per layer in parameter space, whereas we solve the Gauss-Newton system exactly on the batch in sample space by exploiting the least-squares form of the PINN loss.

Research Program

We would build a weighting scheme that trains the network with Adam as usual and, every $T$ steps, computes the hypergradient above. The weights are kept positive and normalized through a softmax parameterization, so that their overall scale cannot trade off against the learning rate.

Each hypergradient costs a fixed set of linear-algebra operations. Forward-mode products give $J\nabla_\theta F$ and $J\nabla_\theta L_j$ for each of the $K$ terms. A conjugate-gradient solve in the $N$-dimensional sample space then costs one forward and one reverse Jacobian product per iteration. A cheaper variant forms $JJ^{\top}$ explicitly for small batches. No step ever differentiates the PDE residual twice with respect to $\theta$. That is the property that makes Hessian-vector products of PINN losses expensive, and it is what our construction removes.

On networks small enough that the exact Hessian can be formed (a few thousand parameters), we would compute a reference hypergradient by exact solve. We then measure the cosine similarity between this reference and each cheap variant across training: identity truncation, Kronecker-factored, penalty/value-function-implied directions, and Gauss-Newton–Woodbury. We relate the gap to the condition number of $JJ^{\top}$, expecting the identity approximation to degrade as stiffness grows. For inverse problems with sparse measurements, the outer objective becomes held-out data misfit plus the log terms, which gives a genuine validation signal.

Baselines

The naive reference is a PINN with fixed unit weights. On Helmholtz, Wang et al. report roughly 15% relative $L^2$ error for this setup.

The strongest existing accuracy baseline is learning-rate annealing from the same paper. Combined with their multiplicative architecture, it reaches 0.1% relative $L^2$ error on Helmholtz. Because that figure mixes weighting with architecture, we would run every weighting method on that same architecture and also on a plain multilayer perceptron, so that the weighting effect is isolated. Beating it means lower median error over at least five seeds at matched wall-clock time.

The strongest efficiency baseline is ReLoBRaLo. It is reported as about 40% faster than annealing and 70% faster than GradNorm on Burgers. No reported error number for it was found on Lacuna, so we would rerun it with its own recommended settings plus an equal-size hyperparameter search.

Inverse-Dirichlet weighting (Maddu et al.) is included on active turbulence. There, convergence order 4 against order 1 is reported, but no absolute error figure was found on Lacuna.

The key ablation sets $G=I$ while keeping our outer objective. This isolates curvature as the ingredient. A second ablation swaps our log objective for LDC-MTL's loss-gap objective while keeping Gauss-Newton curvature. We also run LDC-MTL itself, BOME, and F2SA as off-the-shelf inversion-free bilevel weighters, and a Kronecker-factored hypergradient following Liao et al. All methods get the same collocation points, iteration budget, and a fixed tuning budget of 20 configurations each.

Why not just…?

Why not just use ReLoBRaLo or annealing, which are cheap and already work? They may well be good enough on easy problems, and we expect a tie there. The evidence for a gap is indirect. Loss-ratio rules cannot see opposing gradient directions, which Luo et al. show persist when losses look balanced. Magnitude rules ignore direction altogether. The experiment that settles it is the Helmholtz frequency sweep at matched wall-clock time. If our method does not widen its lead as stiffness rises, the heuristics win and we would say so.

Why not just apply LDC-MTL, which is already a first-order bilevel balancer with constant cost? Its constant cost comes from dropping curvature, and its objective equalizes loss values. Both choices are exactly what PINN pathologies are suspected to defeat, but that suspicion is untested. Both ablations above, together with the cosine-similarity diagnostic, test it directly. On CelebA, where LDC-MTL reports a performance drop of −1.31% against FairGrad's 0.37%, we expect the truncated and curvature-aware hypergradients to agree closely. If they also agree on Helmholtz, the curvature story fails.

Why not just use BOME or a penalty method, which avoid the Hessian entirely? These avoid inversion by turning inner optimality into a constraint or penalty, as Mehra and Hamm and BOME do. That adds a penalty schedule which, per the moving-target concerns in this literature, becomes numerically stiff exactly when the inner problem is stiff. We run them head-to-head. Their failure or success on PINNs is itself an untested result worth reporting.

Why not just train with a Gauss-Newton optimizer and drop weighting altogether? This is the most serious threat. When the batch Jacobian has full row rank and damping is small, the Woodbury form shows that the Gauss-Newton step barely depends on $W$. In that regime, loss weights stop mattering. Weighting matters only where the residual kernel is ill-conditioned and damping is large, which is plausibly the stiff regime but must be shown. We include a damped Gauss-Newton optimizer with unit weights, run at the same wall-clock budget. Our method uses the solve only every $T$ steps, so it is far cheaper per step. If full Gauss-Newton training beats curvature-aware weighting at equal time, the honest conclusion is that curvature belongs in the optimizer, not in the weights.

Proposed Experiments

The main suite uses the benchmarks in this literature: 2D Helmholtz with increasing wavenumber, 1D Burgers with four loss terms, nonlinear Klein-Gordon, the Kirchhoff plate (many boundary terms), 2D Poisson with varying wavenumber, and steady Navier-Stokes. Active-turbulence coefficient estimation serves as the inverse problem with sparse data. Metrics are relative $L^2$ solution error, per-term held-out residuals, wall-clock time and peak memory per hypergradient, and the fraction of training steps with negative inner products between term gradients, as measured by Luo et al.

The first decisive check is the small-network exact-Hessian study. It succeeds if the Gauss-Newton hypergradient stays close to the exact one (cosine above roughly 0.9) while the identity approximation drops well below it on high-wavenumber Helmholtz but not on CelebA.

The second is end-to-end accuracy. It succeeds if curvature-aware weighting beats both annealing and ReLoBRaLo at equal wall-clock time on the stiffest settings, and the identity-truncated ablation does not.

The third is the Gauss-Newton optimizer comparison, which determines whether the benefit lives in the weights at all.

A negative result on the first check would be informative by itself. It would mean the cheap truncations used by LDC-MTL are safe for scientific machine learning, and the bilevel route to PINN weighting reduces to choosing a good outer objective.