Some neural networks for time-dependent physics let their weights change with time, $W = W(t)$, instead of feeding time in as an extra input. These dynamic-weight networks solve hard equations with very few parameters. The best current example, TINNs, trains fewer than 2,000 parameters, and training is only stable under Levenberg-Marquardt (LM), a second-order least-squares method. LM builds and factorizes a curvature matrix from the Jacobian at every step, and that cost is what keeps these models tiny. This proposal exploits a structural fact about such models. Their curvature matrix is a sum of contributions from separate time windows, and during training those contributions go stale at very different rates. Windows where a shock is forming change quickly. Smooth windows barely change. We propose to cache each window's curvature contribution and recompute only the windows whose piece of the weight trajectory has changed. Staleness is judged by comparing a short summary of the low-dimensional code that generates the weights over each window against the same summary at the last refresh.
The summary we test is the path signature. It comes from rough path theory and converts a path into a fixed-size list of iterated integrals: net displacement, signed areas between channels, and higher-order twisting. The contribution we expect to survive is time-blocked lazy curvature for dynamic-weight networks, which would let LM-trained models grow by roughly an order of magnitude. Whether signatures beat a plain distance as the staleness test is a question the experiments must settle, not a premise.
Core Claim
Consider a TINN-style model in which a small subnetwork maps time $t$ to a code $\Phi(t) \in \mathbb{R}^k$, with $k$ around 8 to 16. An affine lift then maps the code to the weights of the spatial network. Split the time domain into windows $\tau_1,\dots,\tau_B$. The Gauss-Newton curvature matrix then decomposes as
$$G = \sum_{b=1}^{B} J_b^\top J_b ,$$
where $J_b$ is the Jacobian of the residuals at collocation points in window $\tau_b$ with respect to all trainable parameters. We claim three things. First, the rows of $J_b$ at a fixed collocation point depend on trainable parameters mainly through the code path $\Phi$ restricted to $\tau_b$. Second, during training the drift of $J_b^\top J_b$ is concentrated in a few windows. Third, a statistic computed only from the $k$-dimensional code path identifies those windows cheaply enough that recomputing only them preserves LM's stability at a fraction of its cost.
The candidate statistic is a signature distance,
$$D_b = \left\| S_{\le N}\big(\Phi|_{\tau_b}\big) - S_{\le N}\big(\Phi^{\text{snap}}|_{\tau_b}\big) \right\| ,$$
where $S_{\le N}$ is the signature truncated at level $N$ (2 or 3), and $\Phi^{\text{snap}}$ is the code path at the iteration when block $b$ was last recomputed. Block $b$ is recomputed when $D_b$ exceeds a threshold. The replacement enters the factorization as a rank-$r_b$ update, where $r_b$ is the number of residual rows in the window.
Because $k$ is small, a level-2 or level-3 signature of the time-augmented code path has at most a few thousand terms. That sidesteps the exponential growth in signature size that restricts signatures to low-dimensional signals.
There is also a specific hypothesis for why signatures might beat a pointwise distance. When a shock shifts slightly in time inside a window, the pointwise change in $\Phi$ is large, while the window's summed curvature changes much less. The signature without a time channel is blind to how fast the path is traversed (Tóth et al.). Its level-2 terms record the order in which code channels move (Kormilitzin et al.). Together these properties should produce fewer false alarms. The hypothesis is falsifiable, and we treat it that way.
Position Relative to Prior Work
Three papers motivate the proposal by exposing the bottleneck.
- TINNs (TINNs) shows that time-varying weights fix the failure of shared static weights across smooth and shock regimes. It also shows that LM is needed to avoid loss spikes. It computes full curvature every step and says this limits the backbone size.
- CoLoRA (CoLoRA) parameterizes weights as time-continuous low-rank updates.
- KoNODE (KoNODE) drives parameters with a linear latent system.
Neither CoLoRA nor KoNODE touches second-order cost. Both are natural later hosts for the method, because both also produce weights from a low-dimensional time-indexed code.
Two lines of work supply the method.
- Lazy curvature. Lazy Hessians reuses a Hessian snapshot for $m$ steps with fresh gradients. GRLM reuses a Gram-matrix SVD so that intermediate LM steps cost $O(d^2)$. Both refresh the whole matrix on a fixed schedule and have only been tested on static-parameter problems. We keep their stale-factor machinery and replace "refresh everything every $m$ steps" with "refresh the stale time blocks."
- Adaptive staleness control. FOAM measures drift in Shampoo's preconditioner using an operator-error proxy on small-eigenvalue directions. It cut eigendecomposition calls to about 10% and 5% of a fixed schedule for the left and right factors. That is our strongest evidence that adaptive triggers pay off. FOAM, however, works on the full matrix and has no notion of time blocks.
Other work either constrains the design or lies outside it.
- SNOpt gathers curvature along a Neural ODE's depth trajectory in one backward pass. It recomputes at every step and assumes Kronecker structure. It shows that trajectory structure can be used for curvature, but not how to amortize curvature over optimization time.
- Marion shows that generalization of continuous-depth models is governed by the Lipschitz smoothness of $W(t)$. This constrains us: if the trigger lets the code path get rough, we may trade speed for accuracy. We therefore monitor held-out error, not only training loss.
- Random CDEs is the closest signature-meets-dynamics work, but it freezes the weights and fits only a linear readout. Signatures there describe inputs. Here they describe the model's own weight path during training, which to our knowledge is new.
Research Program
We first build the instrument. The TINN architecture gets a block-cached Gram matrix, and a rank-$r$ Cholesky update or downdate replaces each refreshed block. Between block refreshes we take GRLM-style intermediate steps against the stale factorization. The LM damping still adapts through the usual ratio of actual to predicted loss reduction, and damping inflation remains available as FOAM-style staleness absorption.
We then scale the spatial backbone from about 2,000 to about 20,000 parameters. At 20,000 parameters a dense factorization is still feasible on one GPU, but per-step full LM is no longer. This is the regime the method is meant to open.
We then run the diagnostic study that decides the method's design. At regular checkpoints we compute every block's true drift $\|J_b^\top J_b - (J_b^\top J_b)^{\text{snap}}\|$ in full. We then measure how well each candidate statistic ranks the windows by true drift. The candidates are:
- signature distance with a time channel;
- signature distance without a time channel;
- level-1 signature only, which is net displacement;
- the sup-norm change of $\Phi$ on the window;
- the change in per-window residual loss.
These measurements are cheap relative to training. They tell us which trigger to use before we spend compute on end-to-end comparisons.
Baselines
The naive reference point is a standard space-time PINN with time concatenated to the input, trained with Adam. It is the default practice, and TINNs reports its time-entanglement failures on Burgers. It gets the same collocation points and the same wall-clock budget. Its learning rate is tuned over the same number of trials as our threshold.
The strongest existing method is TINNs with full LM at every step. It is the only dynamic-weight method in the file reported to beat heavily tuned first-order training. The TINNs paper reports up to a 4× reduction in relative $L^2$ error and reaching target accuracy 10.55× faster than PirateNet with SOAP on Burgers. The file gives these only as ratios: there is no reported absolute relative-$L^2$ value on Lacuna for TINNs on Burgers or Allen-Cahn. We will therefore rerun TINNs ourselves on identical data, at both its original size and our enlarged size, with its published settings. Beating it means one of two outcomes. At equal wall-clock, we reach lower held-out relative $L^2$ error at the enlarged size. At the original size, we reach TINNs' own final error in substantially less time with no increase in loss spikes.
The strongest amortization competitor is GRLM with a global fixed-cadence snapshot. We tune its cadence $m$ over a grid equal in size to our threshold grid. A FOAM-style variant ports FOAM's adaptive damping and refresh-on-threshold rule to the whole LM Gram matrix, without time blocks. We also run Adam and SOAP-style first-order training on the enlarged TINN architecture. Beating the amortization competitors means reaching the same error with fewer Jacobian-row evaluations and less wall-clock time.
Two ablations isolate the key ingredients. The first is time-blocked refresh with a round-robin schedule, which isolates blocking from triggering. The second is time-blocked refresh triggered by sup-norm or displacement change, which isolates the signature itself.
Why not just…?
Why not just use GRLM's fixed cadence? A fixed $m$ must be set for the worst window. The known failure of fixed cadences is instability when the optimizer crosses sharply curved regions such as shock formation. So either $m$ is small and smooth windows are recomputed wastefully, or $m$ is large and shock windows go stale. The diagnostic study measures directly whether drift is concentrated in a few windows. If it is spread evenly across windows, blocking gains nothing and we will say so.
Why not just rely on LM's own damping and loss-ratio test? Damping reacts after a bad step and inflates regularization in every direction at once, which slows progress everywhere. It also cannot say which part of the matrix is stale. We keep damping as a safety net. The experiment that settles this compares the FOAM-style whole-matrix variant against block refresh at equal Jacobian budget.
Why not just train a larger dynamic-weight model with Adam or SOAP? TINNs reports that first-order training of these models produces loss spikes between the initial, boundary and residual terms, and that TINNs beats PirateNet with SOAP by 10.55× in time to target on Burgers. The first-order runs on the enlarged architecture test whether that gap persists at 20,000 parameters.
Why not just use the pointwise change of the code instead of a signature? This is the real threat. The Jacobian depends on the code pointwise, so a sup-norm trigger may do just as well. The diagnostic ranking study and the displacement-trigger ablation decide the question. If signatures do not reduce false refreshes, particularly in windows where a feature shifts in time, we drop them and report time-blocked lazy LM with a pointwise trigger as the contribution.
Proposed Experiments
Benchmarks. We use the TINNs suite: 1D viscous Burgers, Allen-Cahn, KdV and Kuramoto-Sivashinsky. Burgers reference solutions come from PDEBench. We add 2D Burgers as the stress test where the enlarged backbone should matter most.
Metrics. We report held-out relative $L^2$ error against wall-clock time and against Jacobian-row evaluations. We also report the number of loss spikes and the fraction of blocks refreshed per iteration, broken down by time window.
Decisive checks. Four results decide the proposal.
- The signature trigger should rank windows by true curvature drift better than the displacement and pointwise triggers.
- Refreshes should concentrate in shock-forming windows on Burgers and in interface windows on Allen-Cahn. Uniform refresh would falsify the premise.
- At 20,000 parameters, block-lazy LM should beat full LM at equal wall-clock, and should not produce rougher code paths than full LM. Marion's result makes the smoothness of $W(t)$ the variable to watch for generalization.
- On Kuramoto-Sivashinsky, where curvature may drift everywhere, we expect the method to fall back to near-full refresh. That would be an honest boundary on where it helps.
References
- TINNs: Time-Induced Neural Networks for Solving Time-Dependent PDEs
- Bayesian Learning from Sequential Data using Gaussian Processes with Signature Covariances
- An efficient representation of chronological events in medical texts
- CoLoRA: Continuous low-rank adaptation for reduced implicit neural modeling of parameterized partial differential equations
- KoNODE: Koopman-Driven Neural Ordinary Differential Equations with Evolving Parameters for Time Series Analysis
- Second-Order Optimization with Lazy Hessians
- An Enhanced Levenberg-Marquardt Method via Gram Reduction
- FOAM: Frequency and Operator-Error Based Adaptive Damping Method for Reducing Staleness-Oriented Error for Shampoo
- Second-Order Neural ODE Optimizer
- Generalization bounds for neural ordinary differential equations and deep residual networks
- Random Controlled Differential Equations
- PDEBench: An Extensive Benchmark for Scientific Machine Learning