Contractive Reservoir Critics for Learning Chaotic Attractors

When a neural network learns to simulate a chaotic system such as a turbulent flow or the Lorenz equations, tiny errors grow exponentially. Training it to match trajectories point by point therefore teaches it to predict a blurry average, or lets it drift off to infinity. The field's response is to match the long-run statistics of the system, its "climate" rather than its "weather". That requires a way to compare the distribution of generated trajectories against real ones. The two common options each fail in a known way. Fixed-kernel discrepancies such as maximum mean discrepancy (MMD) can be blind to the features that matter. Learned adversarial critics, networks trained to tell real samples from fake, have a smoothness constraint that is enforced only loosely. We propose a third option: a critic built around a reservoir, meaning a large, randomly initialized, fixed recurrent network. Its contraction bounds the critic's smoothness by construction. A small trainable 1D convolutional filter in front of the reservoir learns what to look at.

In two sentences: the critic is "learned convolutional filter, then fixed contractive reservoir, then a small norm-bounded head", so its Lipschitz constant (the maximum rate its output can change as its input changes) is guaranteed by the architecture rather than by a gradient penalty. We train chaotic surrogates against this critic and test whether the learned filter fixes the known blind spot of purely random reservoir features, at the cost of neither stability nor compute.

Core Claim

A Wasserstein-1 critic has to be 1-Lipschitz. The Wasserstein-1 distance is the minimal average cost of moving one distribution's mass onto another's. A Lipschitz bound is exactly what a contractive echo state network supplies over time. Take a critic of the form $f(x) = g(h_T)$, with reservoir state $h_t = \tanh(W h_{t-1} + U c_t)$ and features $c_t$ produced by a causal 1D convolution of the input window $x$. Suppose the recurrent matrix has operator norm (not merely spectral radius) $\|W\| \le \rho < 1$, the convolution kernels are spectrally normalized to Lipschitz constant $L_c$, and the head $g$ is spectrally normalized to $L_g$. Then

$$|f(x) - f(y)| \;\le\; C \sum_{k \ge 0} \rho^{k}\, \|x_{T-k} - y_{T-k}\|, \qquad C = L_g \,\|U\|\, L_c\, \kappa_K,$$

where $\kappa_K$ depends only on the kernel width $K$. The bound follows because $\tanh$ is 1-Lipschitz, so each step back in time multiplies sensitivity by at most $\rho$. The critic is therefore Lipschitz everywhere, not just on the line segments between real and fake samples where WGAN-GP applies its penalty. Its geometry is a fading-memory weighted norm of the kind Differentiable reservoir computing uses to analyze reservoirs on unbounded inputs. Maximizing $\mathbb{E}_{\text{real}} f - \mathbb{E}_{\text{gen}} f$ over the filter and head, with the reservoir frozen, gives a lower bound on a Wasserstein-1 distance between trajectory-window distributions under this weighted norm.

This design also carries a guarantee about sensitivity to initial conditions. The critic is contractive, so it cannot itself amplify small input differences exponentially. Any gradient explosion must come from the generator's rollout, which we control with window length.

The design addresses a specific gap. Transport in reservoir computing proves that contractive reservoirs map input distributions to output distributions continuously in Wasserstein-1. The same paper notes that stability does not imply the reservoir separates distinct input distributions. A random reservoir can be stably blind. Our trainable convolutional front end is the minimal adaptive component meant to supply that separation. We keep the property that makes the reservoir useful here, which is a certified Lipschitz bound, and we add no backpropagation through a trained recurrence inside the critic.

The linear case shows why the front end and a nonlinear head are both needed. If only a linear head $w$ with $\|w\| \le 1$ were trained, the supremum would collapse to the closed-form $\|\mathbb{E} h_{\text{real}} - \mathbb{E} h_{\text{gen}}\|$, a mean-embedding distance close in spirit to RS-W1 from Universal randomised signatures for generative time series modelling. The adversarial content of our method lives entirely in the learned filter and the nonlinear head. That makes the key ablation clean: freeze the filter, use a linear head, and the method reduces to a fixed-reservoir, non-adversarial distance.

Relation to Prior Work

Three works motivate the problem by showing that matching point-wise trajectories fails on chaos and that distribution matching helps. Training neural operators to preserve invariant measures of chaotic attractors adds a Sinkhorn optimal-transport loss on hand-picked physical statistics, or a contrastive embedding. DySLIM adds an MMD penalty with fixed Gaussian kernels. Learning Chaotic Dynamics in Dissipative Systems adds Sobolev and dissipativity terms. All three train the surrogate the same way we will. They differ from us in the distance: theirs is either fixed (MMD, Sinkhorn on chosen statistics) or learned without any Lipschitz guarantee (contrastive features). We learn the distance with a guarantee.

The strongest non-distributional competitor constrains our claims. When are dynamical systems learned from time series data statistically accurate? shows that matching Jacobians gets correct statistics without any distributional loss. That result means a distributional critic is not necessary when Jacobian data exist. Our claim is therefore restricted to settings where Jacobians are unavailable or too costly to compute.

Two papers supply the method. The reservoir theory comes from Transport in reservoir computing and Differentiable reservoir computing. The 1D-convolutional adversarial recipe comes from Adversarial Audio Synthesis, which trains WaveGAN, a 1D CNN generator with a WGAN-GP critic, and introduces phase shuffle so the critic cannot exploit upsampling artifacts.

Our closest relative is Universal randomised signatures for generative time series modelling. It uses a fixed random reservoir (randomized signatures) to define a non-adversarial Wasserstein-type loss, RS-W1, and applies it to financial series. It argues that adversarial training is unnecessary. We take that argument seriously and test it directly on deterministic chaos, which that paper does not address.

Research Program

The surrogate stays fixed across all loss variants, so any difference comes from the loss. It is a small MLP for Lorenz-63, and a 1D convolutional or Fourier neural operator backbone for Kuramoto-Sivashinsky (KS) and Lorenz-96. Every variant uses the same short-horizon MSE term, following the DySLIM loss structure of "short-term term plus $\lambda$ times a distribution discrepancy". The discrepancy is estimated on windows cut from generated rollouts. Gradients flow through the generator only within windows of a few Lyapunov times, and a stop-gradient burn-in lets samples reach the attractor first.

The critic has three parts. First, a causal 1D convolution over time, which for KS also convolves over space. Second, a reservoir of a few hundred to a few thousand units whose operator norm is set by rescaling singular values. Third, a two-layer spectrally normalized head. The critic takes a few steps per generator step, and we report that ratio and its cost.

The work also includes an analysis. We will state and check the Lipschitz certificate above. We will then characterize when a fixed reservoir fails to separate two window distributions: for example, distributions that differ in time-reversal asymmetry or in high-frequency content that the reservoir's input projection averages out. We will show whether the learned filter restores separation.

The memory scale $\rho$ is a physical knob. The critic's effective memory, about $1/(1-\rho)$ steps, can be matched to, shorter than, or longer than the system's Lyapunov time. We expect best results near a match. Beyond about one Lyapunov time the real and generated trajectories decorrelate, so older history adds variance to the distance without adding signal.

Baselines

The naive reference is the same surrogate trained with MSE only, one-step and with unrolled curriculum. The cited literature documents this failure. On Lorenz-63, an MSE model has Wasserstein distributional error 18.97 and a stable Lyapunov exponent near −4.8 against a true −14.5 (When are dynamical systems learned…).

The classical reservoir baseline is an autonomous echo state network: ridge-regression readout, then run closed loop. It is the cheapest serious competitor. We found no reported result on Lacuna for closed-loop ESN invariant-measure error on these benchmarks, so we will run it ourselves. Its reservoir size and ridge strength get the same tuning budget as the other methods.

The strongest existing distribution-matching loss is DySLIM's MMD. DySLIM reports improving the Kolmogorov-flow Time Correlation Metric from 5.39 to 2.45 in the one-step regime, and stable indefinite KS rollouts where curriculum training diverged. We also run the Sinkhorn/optimal-transport variant from Training neural operators…, which reports over 60% reduction in energy-spectrum and histogram error relative to RMSE baselines. Both use our backbone and window protocol.

For absolute reference points on the strongest method per benchmark, we quote the file's figures. On Lorenz-63, Jacobian matching reaches Wasserstein error 0.68 (When are dynamical systems learned…). On 128-dimensional KS, the reported KL divergences are 9.37 for PFNN, 24.53 for MNO, and 142.73 for FNO (Learning Chaos In A Linear Way). These come from different implementations, so we compare against our reruns and treat the quoted values only as sanity anchors.

Two ablations isolate the key ingredient. The first is the fixed-reservoir distance: frozen random filter and linear head, which is RS-W1-like. The second is an unconstrained 1D CNN critic with WGAN-GP and phase shuffle, which is WaveGAN-style and removes the reservoir.

Fairness rules apply to all methods. Every method gets the same data and generator, the same wall-clock budget including critic steps, and the same number of hyperparameter trials, including $\lambda$ for every distributional method. Results are averaged over seeds.

The bar for beating the baselines has two parts. Against DySLIM and the fixed-reservoir distance, we need a lower invariant-measure error (Wasserstein distance on Lorenz-63, KL and energy-spectrum error on KS) at matched compute, with confidence intervals across seeds that do not overlap. Against WGAN-GP, we need equal or better accuracy with visibly lower run-to-run variance and no diverged runs.

Why not just…?

Why not just fit an echo state network with ridge regression and run it closed loop? It is extremely cheap, and it may well reproduce the Lorenz-63 attractor. Ridge regression, however, minimizes one-step $L_2$ error, the same objective the file links to dissipative drift. Transport in reservoir computing guarantees stability of the output distribution but not fidelity. If the closed-loop ESN matches our method on KS and Lorenz-96 statistics, the critic is unnecessary for low-dimensional systems, and we will report that. Our expectation is that the gap opens with dimension.

Why not just use MMD as DySLIM does? MMD is closed form and has a strong record. Its fixed Gaussian kernel can underweight localized, high-frequency structure such as KS shocks. DySLIM itself argues against optimal transport on robustness grounds, so it is a genuine competitor. The head-to-head at equal budget settles it, along with a targeted test on KS energy spectra at high wavenumbers, where a learned convolutional filter should help if it helps anywhere.

Why not just use a fixed random reservoir distance with no adversary (RS-W1)? This is the most serious threat, because it keeps our Lipschitz guarantee and removes the min-max game entirely. The only reason to prefer our method is the non-separation problem that Transport in reservoir computing leaves open. We test that problem directly with constructed distribution pairs that a random reservoir cannot distinguish, and then on the benchmarks. If the frozen variant matches the learned one, the correct conclusion is that RS-W1 transfers to chaos, which is itself a new result, and the adversarial filter should be dropped.

Why not just use a WaveGAN-style convolutional critic with gradient penalty? Adversarial Audio Synthesis shows that such a critic works well on raw 1D signals. Its Lipschitz control holds only along interpolation lines, and it has no built-in memory scale tied to the Lyapunov time. We will measure its empirical Lipschitz constant on off-interpolation inputs and its failure rate across seeds. If it matches our method on both, the reservoir buys nothing.

Decisive Checks

The first check tests the certificate: the empirical gradient norm of the reservoir critic, probed on random and adversarially chosen windows, must stay below the stated bound, while the WGAN-GP critic's should exceed 1 off the interpolation lines.

The second check tests the claim about blind spots: on constructed pairs, the frozen reservoir's distance should be near zero while the learned-filter critic's is clearly positive.

The third check is the benchmark comparison. On Lorenz-63, 128-dimensional KS, and Lorenz-96 (dimension 40), the learned reservoir critic must beat both DySLIM and the frozen variant on invariant-measure error and on the leading Lyapunov exponents at matched compute. We will also sweep $\rho$ and expect a peak when critic memory is roughly one Lyapunov time.

If the second check fails, the adversarial component is unjustified. If the third check fails against the frozen variant, we will publish the negative result together with the positive finding that a fixed reservoir distance suffices for chaos.