Neural emulators of physical systems are trained to predict the next state of a simulation. They are then run on their own outputs for hundreds or thousands of steps, and small errors snowball until the simulated fluid blurs or blows up. A cheap way to give such an emulator memory is temporal channel shifting, a trick from video models that copies a slice of a layer's feature channels from the previous time step into the current one, at zero parameters and zero arithmetic. In an emulator run step by step, those copied channels form a feedback loop through the model's hidden features. The loop can carry useful history, and it can also amplify drift. Whether it helps depends on how many channels are shifted and in which layers, and the only reliable way to find out today is to train each configuration with an expensive multi-step loss and roll it out for thousands of steps.
We propose to make that choice cheap. After only a short, ordinary one-step training run, we measure how fast the emulator amplifies a small perturbation along a short rollout, compared with how fast the true physics amplifies the same perturbation. The comparison uses a differentiable reference solver. We test whether this "excess amplification" score ranks shift configurations by their long-horizon stability, and whether it does so better than one-step validation error, standard training-free architecture scores, and short multi-step training runs given equal compute. If it does, the search over memory configurations becomes affordable, and we learn which configurations survive long rollouts.
Core Claim
The claim has two parts, and each can fail on its own. The first part is that causal temporal shifting of hidden channels across rollout steps is a meaningful, zero-parameter source of memory for autoregressive PDE emulators. Its configuration changes time-to-divergence by a large factor, so the choice is worth searching over. The second part is that a perturbation-growth score computed on briefly trained models predicts the long-horizon ranking of these configurations. Here "briefly trained" means a small fraction of full training, with no backpropagation through long rollouts. The score should do this with higher rank correlation per GPU-hour than the obvious low-fidelity alternative, which is to train each candidate briefly with a multi-step loss and measure its short-rollout error.
The score is defined on the augmented state $z_t = (s_t, c_t)$, where $s_t$ is the physical state and $c_t$ is the cache of shifted hidden channels carried from the previous step. On states drawn from true trajectories, we estimate the $k$-step growth of the leading perturbation direction using Jacobian-vector products and power iteration:
$$ A_\theta = \frac{1}{k}\log \frac{\|J^{(k)}_\theta v\|}{\|v\|} \;-\; \frac{1}{k}\log \frac{\|J^{(k)}_{\text{ref}} u\|}{\|u\|}. $$
Here $J^{(k)}_\theta$ is the Jacobian of $k$ emulator steps with respect to the augmented state, $J^{(k)}_{\text{ref}}$ is the Jacobian of $k$ reference-solver steps with respect to the physical state, and $v, u$ are their leading directions found by power iteration. A chaotic system must amplify some perturbations, so raw growth alone is not a useful signal. What matters is growth beyond what the physics itself produces. We also compute a spectrally resolved version of $A_\theta$ by restricting perturbations to Fourier bands, because divergence often starts as energy piling up at high wavenumbers.
Research Program
The motivating evidence comes from four sources. APEBench built a suite of 46 procedurally generated PDEs with a differentiable pseudo-spectral JAX solver. It showed that one-step, unrolled, and "diverted-chain" training give different stability, and it used only fixed hand-designed ResNets and FNOs. We reuse its solver both for ground-truth trajectories and for $J_{\text{ref}}$. Comparing and Contrasting Deep Learning Weather Prediction Backbones found that short-term error is decoupled from long-term stability, and that ConvLSTM's recurrent memory helps over multi-day horizons. That decoupling is why one-step validation error is a weak selection signal, and the ConvLSTM result is why memory deserves attention at all. Towards Stability of Autoregressive Neural Operators traced divergence to high-frequency aliasing and spectral blocking, the pile-up of energy at the grid's resolution limit. That finding motivates the band-restricted score, and it limits our claims: shifting does nothing against aliasing by itself, so we report results with and without its anti-aliasing blocks.
The method comes from the video literature. TSM introduced residual shifting and a causal online variant with a feature cache. Temporal shift reinforcement learning found that the best shift fraction differs by domain, at 1/5 to 1/3 rather than 1/4, and must be hand-tuned. RubiksNet made shifts learnable under a budget for single-pass classification. Our setting differs from all three in one important respect. Those works shift channels within a fixed clip whose frames are all given. We shift channels across emulator steps, so the shifted features are the model's own past outputs, and the shift becomes a recurrent path that is exposed to compounding error.
The architecture-search literature supplies the evaluation standard and a warning. EZNAS evolved training-free scores, reaching a Kendall's tau of 0.65 on NASBench-201 CIFAR-10. It also noted that such scores fail to order the top 10% of candidates. Zero-cost Proxy for Adversarial Robustness Evaluation and Dynamical Isometry based Rigorous Fair NAS tie robustness and fair ranking to Jacobian and curvature statistics at initialization. Our score is closest in spirit to these Jacobian-based scores, but it departs from them on purpose. An untrained emulator's rollout dynamics say nothing about the learned dynamics, so we measure after brief training and compare against the reference physics rather than against an isometry target. We include the at-initialization version as an ablation to test that reasoning directly.
What we build has three pieces. The first is a shift-equipped emulator family on a dilated ResNet and a U-Net backbone. Each block chooses a causal shift fraction from {0, 1/16, 1/8, 1/4, 1/3}, a temporal offset of one or two steps, and residual versus in-place placement. The second is a ground-truth set of about 200 configurations, sampled to cover this space, fully trained with a fixed multi-step recipe and rolled out to long horizons on each task. This set is expensive, but we build it once and release it. The third is the score, together with a simple search loop: score many candidates and fully train only the top few.
Baselines
The naive reference point is a Markov emulator with no memory: the same backbone with all shift fractions set to zero, trained with the identical multi-step recipe. It shows whether shifting adds anything. A second architectural baseline is input frame stacking, which feeds the last two to four physical states as extra input channels, trained identically and matched in parameter count by widening the first layer. This is the most direct competitor to shifting as a memory mechanism. We also include a ConvLSTM backbone, since the backbone comparison study found it strong at medium horizons. The last architectural baseline is a hand-chosen causal shift at 1/4 in every block, following TSM. On the FNO tasks, every architecture is also run with the anti-aliasing blocks of Towards Stability. Reported context: those blocks gave stable ERA5 forecasts to 100 days (400 steps), against 12 days for FourCastNet. All architectural baselines receive the same data, the same unroll horizon, and the same hyperparameter-tuning budget in GPU-hours as our searched model, counting the cost of the search.
For the ranking question, the strongest competitor is budgeted low-fidelity estimation in the style of Rethinking Performance Estimation in NAS. Each candidate is trained briefly with a short multi-step loss and ranked by its short-rollout error. We give it exactly the same GPU-hours per candidate as our score, including our short training run, and we tune its unroll length and rollout length on held-out tasks. We also report one-step validation error, the standard training-free scores (SynFlow, SNIP, GradNorm, NASWOT), and the at-initialization version of $A_\theta$. That last item is the ablation that isolates the brief-training ingredient. A second ablation removes the reference term and keeps only raw emulator growth, which isolates the comparison against the physics. For external guardrails, the reference is Thermalizer. On Kolmogorov flow, its base emulators diverge around step 5000, thermalization extends the stable horizon by up to two orders of magnitude, and its drift classifier alone adds roughly 60% to the cost of each step. No reported result was found on Lacuna for APEBench table values, so APEBench comparisons are run fresh.
Beating the ranking baselines means a higher Kendall's tau against true time-to-divergence and higher precision in the top 10% of the ground-truth set, at equal or lower GPU-hours. Beating the architectural baselines means a longer median time-to-divergence and lower long-horizon energy-spectrum error, without worse one-step error than the Markov baseline at equal total compute.
Why not just…?
Why not just unroll longer on a memoryless model? Longer unrolling is the main stabilizer in APEBench. That paper also shows that longer unrolling can degrade first-step accuracy, and its memory cost grows linearly with the horizon. Unrolling alone also cannot stop the aliasing-driven blow-up described in Towards Stability. Still, this is a real threat to the first part of our claim. Our design settles it by giving the Markov baseline the same unroll horizon and tuning budget as the shift models. If memory adds nothing at equal unroll length, we report that, and the ranking result stands independently.
Why not just stack past frames at the input? This is the most serious alternative. Temporal shift reinforcement learning found that shifting beat frame stacking on Atari (River Raid 41,850.3 versus 34,849.2). But Atari is a different regime, and physical states may carry most of the history a model needs. The parameter-matched frame-stacking baseline is the experiment that settles this question. If it matches searched shifts, the honest conclusion is that latent memory adds nothing beyond observed history for these PDEs.
Why not just train each candidate briefly with a multi-step loss and rank by short-rollout error? This is a cheap, plausible approach, and it may simply work. Its weakness is that short rollouts usually end before divergence begins, because errors grow slowly at first and then suddenly (urban microclimate FNO study). Measuring the leading growth rate directly should expose the eventual divergence earlier. We test this head-to-head at matched compute. If the low-fidelity baseline ranks as well, we drop the score and report the search results using that baseline.
Why not just attach Thermalizer to any emulator? Thermalizer works, but it pays a classifier cost of about 60% per step, plus diffusion steps whenever drift is detected (Thermalizer). The two approaches are complementary rather than exclusive. We therefore report stable horizon per unit of inference cost, and we also test whether a searched emulator triggers Thermalizer's denoising steps less often.
Proposed Experiments
The tasks are APEBench's 1D Kuramoto-Sivashinsky, Burgers, and KdV equations, 2D Gray-Scott, and 2D Kolmogorov flow, with Kolmogorov flow run to 10,000 steps to match Thermalizer's regime. Time-to-divergence is the first step at which normalized RMSE exceeds 1 or the log energy spectrum departs from the reference by more than a fixed tolerance. We report both definitions, so that conclusions do not depend on the threshold.
The first decisive check is the spread of outcomes across the ground-truth set. If shift configurations differ in median time-to-divergence by less than a factor of two, the search is not worth doing. The second check is rank quality. We compare Kendall's tau and top-10% precision of $A_\theta$ against the matched-compute low-fidelity baseline, one-step error, and the training-free scores, on held-out PDEs that the score was not designed on. The third check is the ablation pair: score at initialization versus after brief training, and with versus without the reference term. This tells us which ingredient carries the signal. The fourth check is whether searched configurations show a consistent placement pattern that transfers between PDEs, such as shifts concentrated in deep layers, as RubiksNet observed for video. Finally, we report total search GPU-hours against the cost of building the full ground-truth set, so the claimed saving is measured rather than asserted.
References
- APEBench: A Benchmark for Autoregressive Neural Emulators of PDEs
- Comparing and Contrasting Deep Learning Weather Prediction Backbones on Navier-Stokes and Atmospheric Dynamics
- Towards Stability of Autoregressive Neural Operators
- TSM: Temporal Shift Module for Efficient Video Understanding
- Temporal shift reinforcement learning
- RubiksNet: Learnable 3D-Shift for Efficient Video Action Recognition
- EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring
- Fine-grained Unsupervised Domain Adaptation for Gait Recognition
- Dynamical Isometry based Rigorous Fair Neural Architecture Search
- Rethinking Performance Estimation in Neural Architecture Search
- Thermalizer: Stable autoregressive neural emulation of spatiotemporal chaos
- Fourier neural operator for real-time simulation of 3D dynamic urban microclimate