Neural weather and climate emulators replace expensive physics solvers with networks that predict the next atmospheric state from the current one. To forecast weeks or simulate years, they run that one-step network over and over on their own outputs. Small errors then feed back and grow. Whether a design survives thousands of such steps is decided mostly by architecture, yet nobody can afford to find out cheaply. Spherical Fourier Neural Operators stay stable for a full simulated year while a flat-grid Fourier operator lasts about 25 days (Bonev et al.). Today the only way to learn this about a new design is to train it to convergence and roll it out. Architecture search, which already costs thousands of GPU days for image classifiers, has therefore never been pointed at rollout stability. The one search method for neural operators, FWNO, scores candidates on single-step validation loss only (FWNO).
We propose a cheap test that measures how strongly a briefly trained one-step network amplifies small perturbations of realistic atmospheric states, broken down by spatial scale. The bet is that this "amplification spectrum" ranks architectures by their long-rollout lifetime and spectral fidelity much better than single-step loss or a short rollout at the same compute. If so, it can serve as the scoring function in an architecture search for emulators. The proposal has two products. The first is a ground-truth table of fully trained, fully rolled-out architectures, which the field currently lacks. The second is a proxy whose value can be settled cleanly against that table.
Core Claim
Autoregressive error growth has two parts. One is the bias of a single step, which single-step loss captures. The other is how the learned map stretches whatever error it is handed, which single-step loss does not see. The second part dominates at long horizons, and it is scale-dependent: emulators typically blow up through spurious growth at small scales or blur through excess damping there.
Both failures can be read off the network's local linear response (its Jacobian, the matrix of how each output changes with each input) at states drawn from the data. Let $f_\theta$ be the one-step emulator and $x$ a held-out reanalysis state. Let $\delta_k$ be a random perturbation restricted to spatial-scale band $k$, where bands are spherical-harmonic degree ranges on the sphere or Fourier wavenumber shells on a periodic box. We define the $m$-step band amplification as
$$ a_k^{(m)} = \mathbb{E}_{x,\delta_k}\, \frac{\| P_k\, J_{f_\theta^m}(x)\, \delta_k \|}{\|\delta_k\|}, $$
where $J_{f_\theta^m}$ is the Jacobian of $m$ composed emulator steps, $P_k$ projects onto band $k$, and $m$ is small (1 to 10). Each term costs one forward-mode Jacobian-vector product chained along the emulator's own short trajectory, so the whole score costs only a few hundred network evaluations.
The proxy compares $\log a_k^{(m)}/m$ with a reference growth rate per band. On the fluid benchmarks this reference comes from the true solver's Jacobian. On ERA5 it comes from the empirical ratio of band variance across consecutive time steps. The proxy reports the worst-band excess, which flags blow-up risk, and the worst-band deficit, which flags blurring risk.
Relation to Prior Work
Three lines of work motivate the gap.
- External stabilization. ARROW trains a reinforcement-learning scheduler over 6/12/24-hour step sizes on top of a fixed Transformer (ARROW). Thermalizer adds a diffusion model that pulls drifting states back toward the physical distribution, at roughly 60% extra cost per step plus hand-tuned thresholds (Thermalizer). Pangu-Weather trains four models at different lead times to cut the number of steps (Bi et al.). All three treat the backbone as given. We instead ask which backbones need less patching.
- Training-based remedies. NeuralGCM trains its learned physics inside a differentiable solver across multi-step rollouts (NeuralGCM). This is effective but expensive, and it uses a fixed network design.
- Cheap architecture scoring. EZNAS evolves training-free proxies (EZNAS). BPE finds the smallest training budget that preserves the ranking of candidates (Zheng et al.). We reuse BPE's budget-minimization logic directly.
Those scoring methods were validated only on static classification, where nothing is iterated. EZNAS's own limitations section also notes that it assumes standard convolution blocks. We therefore expect initialization-only proxies to fail here, because stability is a property of the trained map, not of the random-weight network. We test this rather than assume it.
FWNO is the closest prior work. It searches over wavelet operator configurations using single-step loss and reports Navier-Stokes error falling from 3.43% to 2.35%. It never rolls out. Our claim is precisely that its scoring signal is the wrong one for emulators.
Research Program
Design space. We define a design space around SFNO-style operators, where the open trade-offs are concrete. The choices include:
- whether nonlinearities act in the frequency domain (which, per Bonev et al., helps accuracy but breaks rotational symmetry);
- the spectral truncation per layer;
- the residual scaling and normalization placement;
- the filter parameterization;
- depth and width.
We add flat-grid U-Net and dilated ResNet blocks so the space includes the known-unstable designs Thermalizer used.
Ground-truth table. On Kolmogorov flow and two-layer quasi-geostrophic turbulence, a cheap periodic-box adaptation of the space, we train about 200 sampled architectures to convergence. We roll each out to 10,000 steps; the Thermalizer paper reports that standard emulators diverge by step 5000. For each we record three quantities:
- the time to leave the physical regime;
- the kinetic-energy spectrum error at long times;
- the RMSE curve over time.
On WeatherBench ERA5 at 5.625° we do the same for about 40 spherical architectures, rolled out at 6-hour steps to one simulated year. Lifetime there is measured by spectrum collapse or explosion of Z500 and T850, alongside 3- and 5-day RMSE and anomaly correlation (ACC, the correlation between forecast and observed departures from climatology).
Proxy evaluation. Every proxy is computed at a sweep of training budgets: initialization, then 1%, 3%, and 10% of full training. This locates where stability information first appears.
Search. If the proxy holds up, we run an evolutionary search on the ERA5 space at fixed compute. Fitness is single-step skill subject to a proxy stability constraint. We fully train the winners.
Baselines
The naive reference is random selection from the design space, plus the hand-designed SFNO configuration, which is the strongest human design in the file. All proxy baselines get the same candidate set, the same data, and the same per-candidate compute, matched in GPU-seconds including any rollout steps.
Converged single-step validation loss. This is FWNO's reward. It is the strongest conceptual competitor because, if single-step accuracy already predicts lifetime, no new signal is needed. We give it an advantage by using fully trained models. Beating it means a higher Kendall rank correlation with lifetime and spectrum error, computed from far cheaper models.
Low-fidelity rollout error. This is the BPE-style baseline. Each candidate is trained at the same short budget and then rolled out for as many steps as the compute that our Jacobian probe consumes. We tune that rollout length on a held-out fold, mirroring BPE's minimum-budget search. This is the most dangerous baseline, and beating it is the decisive test.
Training-free proxies. We run Synflow, NASWOT, and an EZNAS-style evolved proxy, re-evolved on our table with the same search budget. For reference, the strongest reported number in the file is EZNAS-A's Kendall τ of 0.65 on NASBench-201 CIFAR-10, versus 0.57 for Synflow and NASWOT (EZNAS). No result on rollout-stability ranking was found on Lacuna for any proxy, so the bar is internal. To count as useful, our proxy must exceed the best competitor at matched compute by a margin outside bootstrap confidence intervals over architectures.
Stabilization and search competitors. - The searched architecture is compared with SFNO and with a U-Net plus Thermalizer at matched training compute. We report both rollout lifetime and per-step inference cost, since Thermalizer stabilizes at a recurring price. - At equal search compute, we also compare against a search guided by single-step loss. - On WeatherBench, the published 3-day Z500 RMSE anchors are about 154 m²s⁻² for operational IFS, 626 for a direct CNN, and 936 for persistence (WeatherBench). No reported 5.625° RMSE for ClimODE or SFNO was found on Lacuna, so we retrain both ourselves.
Ablations. - Collapsing all bands into a single Jacobian norm tests whether the scale breakdown matters. - Probing at random inputs instead of data states tests whether on-distribution evaluation matters. - Setting $m=1$ versus $m=10$ tests whether short compositions add information.
Why not just…?
Why not just rank by single-step loss, as FWNO does? Single-step loss may well correlate with lifetime within a narrow family of designs. The flat-grid versus spherical FNO contrast, a 25-day versus one-year lifetime, shows that it cannot be the whole story across families. Our ground-truth table measures this correlation directly. If single-step loss reaches the proxy's rank correlation, we report that the stability axis is redundant, and the table itself becomes the contribution.
Why not just do short rollouts of briefly trained models? This is the real threat. A short rollout already contains the growth signal in principle. Our argument is that over a few steps, measured error is dominated by one-step bias, so growth is hidden until long after the probe ends. The Jacobian probe isolates growth rates directly and per scale. The matched-compute comparison in the baselines settles this. If short rollouts win, we adopt them as the search signal and drop the Jacobian machinery.
Why not just stabilize any model with Thermalizer? It works, with up to 100x longer horizons, but it charges about 60% extra per step forever and needs tuned thresholds. It also assumes a stationary system, which forced climate runs are not. We report it alongside a searched architecture on both lifetime and inference cost. The two approaches are also complementary: a lower-drift backbone should trigger the denoiser less often, which we measure.
Why not just train with multi-step losses, like NeuralGCM? Multi-step training changes the weights, not which design is reachable. Its memory cost grows with rollout length, which is exactly why it cannot be applied to every search candidate. We test whether proxy rankings survive switching the final training of the top candidates to multi-step training. If the ranking reshuffles, the proxy would need to be computed under that training regime instead.
Decisive Checks
The project succeeds or fails on one table: Kendall correlation between each proxy and measured rollout lifetime and long-time spectrum error, across about 200 fluid and about 40 ERA5 architectures, at matched compute.
The proxy must beat both short-rollout error and converged single-step loss on both benchmarks. Its rankings computed at 3% of training must also transfer, meaning that the ordering found on Kolmogorov flow should predict the ordering of corresponding spherical designs on ERA5.
Given that, the search must produce an architecture that outlives hand-designed SFNO at equal 3-day Z500 RMSE, or matches it at lower cost. It must do so without an external stabilizer.
References
- Spherical Fourier Neural Operators: Learning Stable Dynamics on the Sphere
- Generative flow induced neural architecture search: Towards discovering optimal architecture in wavelet neural operator
- ARROW: An Adaptive Rollout and Routing Method for Global Weather Forecasting
- Thermalizer: Stable autoregressive neural emulation of spatiotemporal chaos
- Accurate medium-range global weather forecasting with 3D neural networks
- Neural general circulation models for weather and climate
- EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring
- Rethinking Performance Estimation in Neural Architecture Search
- WeatherBench: A Benchmark Data Set for Data-Driven Weather Forecasting