Reservoir computing forecasts a time series with a recurrent network whose internal weights are drawn at random and then frozen; only a final linear layer is trained, usually by ridge regression on one-step-ahead targets. Theory for these models bounds the one-step prediction error, and practice judges them by something else. To forecast many steps ahead, a practitioner feeds the model's own outputs back in as inputs, a mode called "closed loop," and then asks how long the forecast stays on track and whether the long-run statistics are right. When the model can represent the true dynamics, these two criteria agree: a small one-step error means a good closed-loop forecast. We argue that when the model cannot represent the true system, they come apart. A typical case is a chaotic system whose fast scales are never observed. The best one-step readout for a given random reservoir then defines a closed-loop system whose stability depends on which reservoir was drawn. In that regime more training data can make closed-loop forecasts worse, and the random seed becomes a lottery that one-step validation cannot see.
The claim is testable and has practical consequences. If it holds, existing worst-case risk bounds for reservoirs control the wrong quantity under misspecification. Seed-to-seed variance should then be reported as a first-class result, and the cheap fix of trying a few seeds and keeping the best works only when a seed's success is stable under small perturbations of its weights. We propose to measure all three effects on a benchmark where misspecification can be dialed up and down. We then compare simple selection against a readout fitted to closed-loop behaviour, while still using only closed-form ridge solves.
Core Claim
Write $e_k$ for the closed-loop error after $k$ steps, $\varepsilon$ for the one-step residual of the trained readout, and $L$ for a local Lipschitz constant of the closed-loop map (reservoir plus readout fed back on itself) near the trajectory. The standard error-propagation argument gives
$$ e_k \lesssim \varepsilon \sum_{j=0}^{k-1} L^{j}. $$
Risk bounds for reservoir computing controls the estimation part of $\varepsilon$ at rate $O(1/\sqrt{n})$ in the training length $n$, with high probability over reservoir draws. It explicitly leaves approximation error open, and it says nothing about $L$. Under realizability, $\varepsilon$ shrinks toward the noise level and $L$ inherits the true system's local behaviour. Under misspecification, $\varepsilon$ stalls at a reservoir-specific floor. The readout also converges to a reservoir-specific projection whose closed-loop $L$ can exceed the true system's. That $L$ is a property of the projection, not of the estimation noise, so neither more data nor better one-step fit controls it.
Three predictions follow. First, a closed-loop "dip": with ridge strength fixed or tuned on one-step loss, valid prediction time can peak at moderate $n$ and then decline while one-step mean squared error keeps falling. This would be a temporal analogue of The Dipping Phenomenon, where misspecified classifiers converge confidently to a worse solution. Second, closed-loop quality spreads widely across reservoir seeds while one-step error barely varies, so the rank correlation between them collapses as misspecification grows. Third, the open question that decides what to do about it: whether a seed's closed-loop failure is "robust" or "fragile" in the sense of Smoothed Performance Guarantees for Local Search. That paper shows worst-case bottlenecks either survive or vanish under small random perturbations of the instance.
Relation to Prior Work
The motivating evidence comes from Data-driven prediction of a multi-scale Lorenz 96 chaotic system. That study observes only the 8 slow variables of a three-tier chaotic system, hiding 64 intermediate and 512 fast variables. It reports that a reservoir network forecasts accurately for about 2.3 model time units, against about 0.9 for an LSTM and 0.6 for a feed-forward network. It is a single-pipeline benchmark: it does not report variance over reservoir draws, does not treat the hidden tiers as formal misspecification, and does not vary training length. We reuse its setting and add exactly those three axes.
The theory that constrains us comes from two places. Risk bounds for reservoir computing supplies the one-step guarantee we argue is insufficient. Differentiable reservoir computing proves the contraction conditions behind the echo state property and introduces "persistence," a local stability around specific trajectories even when global contraction fails. Persistence is the right lens for the closed-loop map: we estimate whether each trained readout yields a persistent closed-loop system near the attractor. Neither paper considers the readout-induced feedback system.
On the misspecification side, two results tell us what to expect from the fitted readout. Moulines et al. show that misspecified maximum likelihood for hidden Markov models still converges to the divergence-minimizing parameter, because the process forgets its initial conditions. That is the analogue of our claim that each reservoir's readout converges to a well-defined projection; we add that the projection differs across draws. Domke shows that under structural mismatch, fitting the joint likelihood degrades while fitting the loss the downstream task actually uses does better. This directly motivates our closed-loop-targeted readout.
On the initialization side, Does Knowledge Distillation Really Work? and On Learnable Parameters of Optimal and Suboptimal Deep Learning Models document seed-dependent basins in gradient-trained networks. Reservoirs are often credited with escaping this because the readout problem is convex. Our point is that convexity removes optimizer sensitivity but not sensitivity to the representation: the random draw still fixes the hypothesis class. Spectral-Aware Reservoir Computing replaces randomness with frequency-guided wiring for classification. That design presumes clear periodicity, which broadband chaos with hidden scales lacks, so we treat it as context rather than as a baseline.
Research Program
The core testbed is the three-tier Lorenz 96 system. We control misspecification in two ways, using the coupling coefficients between tiers and the set of observed tiers. One setting observes the slow and intermediate variables together; the hardest observes only the slow ones. A realizable control generates data from a "teacher" echo state network with a known readout and gives the student the same reservoir. In this control the one-step and closed-loop criteria should agree by construction, so any dip or seed lottery there would falsify our mechanism.
For each misspecification level we draw 100 reservoirs at fixed size and spectral radius. We train readouts at training lengths spanning roughly two orders of magnitude. For each trained model we record one-step test mean squared error, valid prediction time, and the distance between the predicted and true marginal distributions of the slow variables. Valid prediction time is the closed-loop time until normalized error crosses a fixed threshold, the criterion used by Chattopadhyay et al. We also record an estimate of the closed-loop local Lipschitz constant, taken from finite-time Lyapunov exponents of the fed-back system along test trajectories.
The fragility study perturbs each reservoir's recurrent matrix by relative Gaussian noise at several scales, retrains the readout, and measures how often closed-loop success or failure persists. Robust failure means that a seed which fails keeps failing in a neighbourhood. Fragile failure means its outcome flips under tiny perturbations, and is then essentially chaotic in the weights.
The method we test is a closed-loop-consistent readout. It alternates two steps. In the first, the system is run in partial closed loop, mixing teacher inputs with the model's own outputs on a schedule. In the second, the readout is refitted by ridge regression on the reservoir states this produces, against the true next values. Every step remains a closed-form solve, so the method keeps the cost advantage that makes reservoirs attractive. The method is meant to target $L$ directly rather than only $\varepsilon$, in the spirit of Domke's task-targeted losses.
Baselines
The naive reference is standard practice: one reservoir seed, ridge strength chosen by one-step validation error, and training on all available data. This is the configuration whose behaviour our first two predictions are about.
The strongest published reference is the reservoir pipeline of Chattopadhyay et al., which reports a valid forecast horizon of about 2.3 model time units (over 10 Lyapunov times) on the slow-only multiscale Lorenz 96 task. Their LSTM, at about 0.9 time units, is a secondary reference. No reported seed variance for this pipeline was found on Lacuna. We therefore re-run their configuration ourselves across 100 seeds and treat our reproduced median as the operative number. We will report it alongside their 2.3 rather than substituting it.
The strongest practical competitor is best-of-K selection: draw K reservoirs, tune ridge strength for each by short closed-loop rollouts on a validation segment, and keep the best. A second competitor injects small noise into the fed-back inputs during training, a common stabilization trick for which we found no reported Lorenz 96 number on Lacuna. A third is an ensemble that averages the closed-loop forecasts of K independently trained reservoirs. A fourth is a single reservoir K times larger, which tests whether richer features alone remove the problem.
Fairness means the same training trajectories and the same validation segment for every method. We also count reservoir simulations and ridge solves as the compute budget, so the closed-loop-consistent readout gets no more total solves than best-of-K selection gets candidates.
The key ablation applies the closed-loop-consistent readout to a single fixed seed. This separates the benefit of changing the readout from the benefit of choosing among draws. Beating the field means a higher median and a higher tenth-percentile valid prediction time than best-of-K selection at matched compute, with a smaller seed-to-seed spread, at the stronger misspecification levels.
Why not just…?
Why not just pick the best of K seeds using closed-loop validation? This is the real threat, and it may be enough. Whether it is depends on the fragility study. If failures are robust, the outcome on the validation segment predicts the outcome on the test segment and on nearby weights, and selection should work. In that case we will report it as the recommended practice, and the contribution becomes the diagnosis rather than the method. If failures are fragile, selection mostly picks lucky noise. Its validation-to-test rank correlation should then be weak, and only a method that lowers $L$ itself will reduce the spread.
Why not just use a bigger reservoir? A larger random reservoir shrinks the approximation floor, but the projection it converges to is still the one-step one, so nothing in the error bound above ties its closed-loop constant to the true system's. The size-matched comparison against K concatenated-scale reservoirs tests this directly. If the large reservoir removes both the dip and the spread, our mechanism is wrong at practical sizes.
Why not just train an LSTM end to end? On this exact benchmark the reported LSTM horizon of about 0.9 time units is well below the reservoir's 2.3 (Chattopadhyay et al.). Gradient-trained recurrent networks also carry their own seed sensitivity (Stanton et al.). We will include LSTM seed variance for completeness, but it is not the competitor that decides the question.
Why not just add training noise or tune ridge strength harder? Both are forms of regularization that may lower $L$ without being designed to. They are included as baselines under the same closed-loop validation budget. If either matches the closed-loop-consistent readout, we will say so.
Decisive Checks
The mechanism is supported if three things hold together. In the realizable teacher-reservoir control, valid prediction time rises monotonically with training length and correlates strongly with one-step error across seeds. Under slow-only observation of Lorenz 96, at least some seeds show valid prediction time falling at large training lengths while one-step error keeps falling. And the across-seed rank correlation between one-step error and valid prediction time falls toward zero as tier coupling increases.
The mechanism is refuted if the dip and the decorrelation also appear in the realizable control, or if they vanish once ridge strength is tuned on one-step loss at every training length. The method question is settled separately, by the matched-compute comparison against best-of-K selection and by the fragility measurement that predicts which of the two should win.
References
- Risk Bounds for Reservoir Computing
- The Dipping Phenomenon
- Smoothed Performance Guarantees for Local Search
- Data-driven prediction of a multi-scale Lorenz 96 chaotic system using a hierarchy of deep learning methods: Reservoir computing, ANN, and RNN-LSTM
- Differentiable reservoir computing
- Asymptotic properties of the maximum likelihood estimation in misspecified hidden Markov models
- Learning Graphical Model Parameters with Approximate Marginal Inference
- Does Knowledge Distillation Really Work?
- On Learnable Parameters of Optimal and Suboptimal Deep Learning Models
- Spectral-Aware Reservoir Computing for Fast and Accurate Time Series Classification