Anytime-Valid Drift Alarms for Autoregressive Emulators

A neural emulator is a network that predicts the next state of a physical system, such as a turbulent flow, and is then fed its own prediction to produce the step after that. Run this way for thousands of steps, small errors compound and the simulated state drifts away from anything the real system could produce. The most effective current fix leaves the emulator alone and adds a watchdog. A monitor decides when the rollout has drifted too far, and a generative model then pulls the state back toward physically plausible configurations. The weak point is the monitor. It fires on a hand-tuned threshold, so nobody can say how often it raises false alarms, how long it takes to notice real drift, or whether a threshold tuned on one system will work on another.

We propose to replace the heuristic threshold with a statistical test that watches the rollout continuously. The test fires only when the evidence of drift is strong enough, and its false-alarm probability over the entire rollout, however long, stays below a user-chosen level $\alpha$. The difficulty is that the clean sequential tests from statistics, called confidence sequences (confidence intervals that stay valid no matter when you look), require observations that lie in a known bounded range. Emulator states do not, and their error grows without bound when the rollout blows up. Our central move is to convert each emulated snapshot into bounded monitoring scores. Each score is a percentile rank of a physical summary, such as kinetic energy, enstrophy or the energy in a band of spatial scales, measured against the system's true long-run distribution. Percentile ranks live in $[0,1]$ by construction and are uniform when the rollout is healthy. Bounded-support sequential tests then apply exactly, and a single $\alpha$ replaces per-dataset threshold tuning.

Core Claim

We claim that turning rollout summaries into percentile ranks and testing them with betting-style confidence sequences gives a drift detector with three properties. Its false alarm rate over any horizon is provably at most $\alpha$, up to a correction for temporal dependence that we measure directly. It detects gradual drift, including the loss of small-scale variance that blurs long forecasts, earlier than pointwise range checks. When it gates a denoising step, it matches a well-tuned heuristic monitor in stable horizon while calling the expensive correction less often, using one $\alpha$ across systems.

The bounded-support requirement motivates the construction rather than being an assumption we hope holds. Distribution-free inference for regression: discrete, continuous, and in between proves that distribution-free confidence intervals of finite width are impossible when the response is unbounded, because rare large outliers can move the mean arbitrarily. That negative result rules out testing raw energies or raw errors directly. The percentile-rank transform avoids it without needing to know any physical bound $B$.

Research Program

We first estimate the invariant measure of each system, meaning its long-run distribution over states. For a set of $k$ summaries $g_1,\dots,g_k$ (total energy, enstrophy, and spectral energy in a few wavenumber bands), we build empirical cumulative distribution functions $F_j$ from long ground-truth simulator runs. For an emulated snapshot $\hat u_t$, the score is $Z_{t,j} = F_j(g_j(\hat u_t))$. When the emulator samples the true invariant measure, each $Z_{t,j}$ is uniform on $[0,1]$, with mean $1/2$ and variance $1/12$.

We then monitor two kinds of departure. A shift in the mean of $Z$ signals bias, such as energy pile-up at small scales, the spectral blocking that Towards Stability of Autoregressive Neural Operators identifies. A drop in the variance of $Z$ signals the collapse toward over-smoothed states that WeatherBench 2 and Fixing the Double Penalty document.

For the mean, we use a betting process. This is a running product of fair bets against the healthy hypothesis:

$$K_t = \prod_{s \le t}\big(1 + \lambda_s (Z_s - \tfrac12)\big).$$

Here $\lambda_s$ is a bet size chosen from past scores only and constrained to keep each factor positive. If the rollout is healthy, $K_t$ is a nonnegative martingale. Ville's inequality then gives that $K_t$ ever exceeds $1/\alpha$ with probability at most $\alpha$, which is where the horizon-free guarantee comes from.

For the variance, we reuse the time-uniform empirical Bernstein confidence sequences of Sharp Empirical Bernstein Inequalities for the Variance of Bounded Random Variables. That paper constructs confidence sequences for the variance of $[0,1]$-valued streams and proves their width matches an oracle constant. We fire when the sequence excludes $1/12$ by more than a tolerance. For the joint $k$-dimensional score vector, we use the multivariate construction of Gambling-Based Confidence Sequences for Bounded Random Vectors, which requires a known compact convex domain. The hypercube $[0,1]^k$ satisfies this by design.

Two honest complications shape the design. First, a trained emulator is never exactly on the invariant measure. A test of an exact null would eventually fire on harmless small bias and over-trigger corrections. We therefore test a composite null that allows the mean to lie within $1/2 \pm \delta$ and the variance within a band around $1/12$, with $\delta$ set from the spread of scores across independent ground-truth runs.

Second, the martingale argument needs each score to have the null mean given the past, but chaotic trajectories are autocorrelated. We thin the stream, scoring only every $\tau$ steps, with $\tau$ set to several decorrelation times estimated from ground truth. We then check empirically how much residual dependence inflates the false alarm rate, by feeding true simulator trajectories to the detector as if they were rollouts. After each alarm and correction, the betting process restarts.

In closed loop, an alarm triggers the reverse-diffusion correction of Thermalizer: Stable autoregressive neural emulation of spatiotemporal chaos. That method trains an unconditional diffusion model on the invariant measure and denoises the drifted state when a classifier-predicted noise level crosses hand-set thresholds. Our work changes only the decision of when to correct. We supply Thermalizer's missing part, a principled trigger, and do not compete with its projection step.

Baselines

The naive reference is the unmonitored emulator, a U-Net or Dilated ResNet trained on one-step loss exactly as in Thermalizer. The paper reports that such an emulator "diverges into non-physical states by step 5000" on Kolmogorov flow. Every closed-loop method must extend this horizon, and every open-loop detector must fire before this point.

The strongest existing method is Thermalizer itself, using its classifier-predicted noise level and thresholds tuned per dataset. The paper reports stable horizons extended "up to 100x in some cases," with the classifier adding roughly 60% to per-step cost. It reports no scalar error values on Kolmogorov flow or quasi-geostrophic turbulence, so we will regenerate its curves with the same emulators, the same diffusion model and the same denoising schedule. To make the comparison fair to Thermalizer, we give its thresholds a grid search on validation rollouts. We give our method no tuning beyond fixing $\alpha = 0.01$ and $\delta$ from ground-truth spread across all systems. Beating it means reaching an equal or longer stable horizon and equal or lower long-run statistical error with fewer denoising calls. Transfer is the sharper test: thresholds tuned on Kolmogorov flow and applied unchanged to quasi-geostrophic turbulence, versus our untouched $\alpha$.

The pointwise range check is the simple competitor we take most seriously. It flags a snapshot whenever any summary falls outside the central $1-\alpha'$ range of the ground-truth distribution, with $\alpha'$ Bonferroni-split over the expected number of checks. Classical CUSUM on the raw energy, a change-point detector that accumulates evidence under a Gaussian pre-change and post-change model, is included with the same thinning. These baselines share our summaries and our ground-truth data, so any difference comes from accumulating bounded evidence rather than from better features.

Three ablations isolate the key ingredients. The first replaces percentile ranks with raw summaries clipped to a guessed physical bound $B$, which tests whether the bounded-by-construction transform matters. The second drops thinning. The third monitors the mean only, dropping the variance test, to measure how much of the benefit comes from detecting over-smoothing.

Why not just…?

Why not just use conformal prediction intervals on the rollout? Valid Error Bars for Neural Weather Models using Conformal Prediction calibrates per-lead-time error bars on a held-out set and reports about 91% coverage at a 90% target. Those intervals answer a different question: how large the error is at step $t$, given that the model behaves as it did during calibration. The authors themselves note that exchangeability can fail under unprecedented shifts. Their guarantee is also marginal per lead time, not simultaneous over a trajectory. We do not claim conformal intervals are wrong. We claim they do not provide a stopping rule with horizon-wide false-alarm control. Our comparison of false-alarm rates over long horizons makes this concrete.

Why not just check whether energy leaves its normal range? This is a real threat. For explosive blow-up, a range check will likely fire almost as fast as our detector. Where it should lose is gradual drift, in which every snapshot looks individually plausible but the distribution has shifted or narrowed. Its per-check false alarms also add up over long horizons unless it is Bonferroni-corrected, which in turn delays detection. We will report injected-drift experiments separated by drift type. If the range check matches us on gradual and variance-collapse drift as well, we will say so. The contribution would then shrink to calibrated thresholds for an existing heuristic.

Why not just train the emulator to be stable, with multi-step training, HERO, or anti-aliasing architectures? HERO: History-Enriched Rollout Training for Long-Horizon Autoregressive Neural Operators penalizes rollouts that repeat a past checkpoint's failure modes, reducing Burgers nRMSE@100 from 0.541 to 0.153. Towards Stability of Autoregressive Neural Operators extends stable ERA5 rollouts from 12 to 100 days. Both delay drift and neither certifies its absence, so a monitor remains useful on top of them. We will run our detector on a HERO-trained emulator on Kolmogorov flow. The prediction is later but still non-zero alarms, with the false-alarm guarantee intact. No reported Kolmogorov-specific scalar for HERO was found on Lacuna.

Proposed Experiments

The benchmarks are 2D Kolmogorov flow and two-layer quasi-geostrophic turbulence, using Thermalizer's emulators. We run open-loop detection first, then closed loop.

The first decisive check is validity. We feed long true-simulator trajectories to every detector at $\alpha = 0.01$ and measure the fraction of trajectories that raise any alarm within horizons up to $10^5$ steps. Our detector passes if it stays at or below $\alpha$ with thinning, and we report how far it degrades without thinning. The range check and CUSUM are measured the same way.

The second check is detection delay. We splice true trajectories into emulator rollouts at known onset times. We also create controlled drifts: small-scale energy injection to mimic blocking, and spectral low-pass damping to mimic over-smoothing. We report the delay from onset to alarm as a function of drift magnitude.

The third check is closed-loop stability. We gate Thermalizer's denoising with each trigger and measure the stable horizon, the number of denoising calls per $10^4$ steps, and the distance between the rollout's long-run energy and enstrophy histograms and ground truth. We run this both with per-system tuning and with cross-system transfer of the tuned thresholds.

As a stretch, we apply the open-loop detector to long ERA5 rollouts of a spectral-normalized emulator following Towards Stability of Autoregressive Neural Operators, using summaries such as global kinetic energy by latitude band. There, seasonal non-stationarity forces the reference distributions $F_j$ to be conditioned on time of year. Whether that conditioning preserves validity is itself an open question the experiment would answer.

The proposal fails if the validity check shows dependence breaking the guarantee even after thinning. It also fails if the pointwise range check matches our delay and call counts across all drift types. Either outcome is informative and would be reported.