Reservoir computing processes time series with a large recurrent network whose weights are drawn at random and never trained. Only a linear readout on top is fit, in closed form by ridge regression. As the reservoir gets wider, its state statistics settle toward a deterministic limit, a fixed "temporal kernel" that measures how similar two input histories are. Recent work uses this limit to choose hyperparameters without running the reservoir at all. Real reservoirs, however, are finite and often small. They are also usually tuned close to the boundary of stable dynamics, which is exactly where infinite-width predictions are least trustworthy.
This note proposes to measure and predict the gap between a finite reservoir and its infinite-width limit for the standard nonlinear (tanh) echo state network. It has two goals: to state the rate at which this gap closes, and to use that rate to choose hyperparameters for the width that will actually be deployed. The central claim has two parts. First, the average test error of a ridge-trained reservoir approaches its infinite-width value at rate $1/N$ in the width $N$. This is faster than the $1/\sqrt{N}$ rate of existing reservoir approximation bounds. Second, the coefficient of that $1/N$ term grows sharply as the spectral radius approaches 1. As a result, the best hyperparameters move as the width changes. A selector that ignores width will therefore pick the wrong settings for small reservoirs.
Core Claim
Let $\theta$ denote the reservoir hyperparameters: spectral radius, input scaling, leak rate and ridge penalty. Let $R(N,\theta)$ be the expected test risk of a width-$N$ tanh echo state network with a ridge readout, where the expectation is over random reservoir draws. In the regime where training length is much larger than width, which is typical for reservoirs, we conjecture
$$R(N,\theta) = R_\infty(\theta) + \frac{a(\theta)}{N} + o(1/N),$$
where $R_\infty(\theta)$ is the risk of kernel ridge regression with the infinite-width temporal kernel. We further conjecture that the seed-to-seed spread of the risk shrinks only as $1/\sqrt{N}$.
The $1/N$ form follows the strong-versus-weak error distinction from mean-field theory, where averaged quantities converge at $O(1/N)$ while individual trajectories converge at $O(1/\sqrt{N})$ (Suzuki et al.). It also matches the $\mathcal{O}(1/N)$ squared deviation and the $R_N \approx R_\infty + c/N$ law observed for trained wide networks in Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networks. In both cases, first-order fluctuations of the random features cancel in expectation.
We also conjecture that $a(\theta)$ scales like a power of $1/(1-\gamma(\theta))$. Here $\gamma(\theta)$ is the contraction rate of the infinite-width kernel recursion: how strongly one step of the limiting covariance map shrinks differences between states. At that boundary, the feedforward Wasserstein bounds of Quantitative Gaussian approximation of randomly initialized deep neural networks diverge.
If both conjectures hold, the practical consequence follows directly. The hyperparameter minimizing $R_\infty$ sits closer to the stability edge than the one minimizing $R(N,\theta)$ at moderate $N$. The two optima differ by a predictable, width-dependent amount.
Relation to Prior Work
Three papers motivate the gap.
Approximation Bounds for Random Neural Networks and Reservoir Systems proves that a readout exists achieving $L^2$ error $C^*/\sqrt{N}$ for ReLU echo state networks. It is an upper bound only. It does not analyze the ridge readout actually fit, and it does not cover tanh reservoirs. Infinite-dimensional reservoir computing obtains $O(\sqrt{\log n/n} + 1/\sqrt{N})$ total error, but only for linear state equations. Risk bounds for reservoir computing explicitly bounds estimation error and leaves approximation error aside. None of the three predicts a $1/N$ average rate, how the constant depends on hyperparameters, or behaviour near the stability edge. Those are our targets.
The closest work is Free-Probability Kernels for Zero-Rollout Hyperparameter Selection in Reservoir Computing. It computes the limiting kernel of a leaky linear reservoir and selects hyperparameters by kernel ridge regression on short pilot sequences, without running the reservoir. Its limitations frame our contribution in two ways. It is restricted to linear recurrence, and it gives no finite-width rate; its finite-width check only shows the gap shrinking between widths 1,000 and 20,000. We differ on both counts. We target the nonlinear tanh recurrence, and our object is the width correction, not the limit.
Our method draws on two sources. For the nonlinear limit we use Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes. That paper shows random RNNs with tied weights converge to Gaussian processes and supplies a recursion for the limiting covariance across time steps. An echo state network is exactly such a random RNN with a trained readout, so this gives a computable tanh-reservoir kernel. Yang's result holds only for a fixed number of time steps, at initialization, and without a readout. Our analysis adds uniform-in-time control from contractivity, plus the ridge resolvent.
Non-asymptotic approximations of neural networks by Gaussian processes acts as a constraint. Distributional convergence to the Gaussian limit can be logarithmically slow for ReLU and sub-polynomial for tanh. We therefore deliberately target the expected ridge risk, a smooth functional of the state covariance, rather than full distributional closeness. We will report where this choice stops working.
Research Program
The work has three parts.
Theory. We first write the infinite-width state-covariance recursion for leaky tanh reservoirs driven by an input sequence. We then show that when the recursion is contractive with rate $\gamma<1$, finite-width covariance errors stay bounded uniformly in time, with a constant of order $1/(1-\gamma)$. This addresses the open problem that errors in recurrent networks might compound over long horizons. Next, we expand the ridge resolvent $(XX^\top+\lambda I)^{-1}$ to second order around its deterministic equivalent. This yields $a(\theta)$ as an expression in the limiting kernel and its fourth-order moments, which can be evaluated by Monte Carlo at modest cost. We expect a fully rigorous proof only for leak rates and gains well inside the contractive region. Near $\gamma\to1$ the theory will make a scaling prediction that is checked empirically.
Measurement. We run large controlled sweeps of width, spectral radius and seeds to test the exponent, the coefficient and the drift of the optimum.
Selector. We build a width-aware hyperparameter selector. It ranks candidates by $\hat R_\infty(\theta) + \hat a(\theta)/N_{\text{deploy}}$, where both terms come from the kernel computation with no full-width rollouts. A cheaper hybrid variant instead estimates $a(\theta)$ from rollouts at two small widths, using the theoretically fixed exponent.
Baselines
The naive reference point is a fixed textbook configuration: spectral radius 0.9, unit input scaling, and a ridge penalty chosen by validation only. It shows how much any selection buys.
The strongest existing method is exhaustive grid search at the deployment width. On the synthetic suite of Free-Probability Kernels for Zero-Rollout Hyperparameter Selection in Reservoir Computing, this baseline (Direct500) reaches a mean deployment score ($1-\text{NRMSE}$) of 0.774. The same paper's zero-rollout selector reaches 0.772. We rerun both on tanh reservoirs with an identical candidate grid, identical pilot and evaluation sequences, and identical seeds. The free-probability selector is run as published, using its linear-recurrence kernel as a proxy for the tanh reservoir; this is the realistic way a practitioner would use it.
Our selector beats these baselines if it meets or exceeds the selection quality of the deployment-width grid search, at no more than the free-probability method's cost, at small deployment widths ($N\le 500$). There we predict that infinite-width selectors lose measurably.
Two ablations isolate the key ingredient. The first is the infinite-width tanh kernel selector without the $1/N$ term, which separates "nonlinear kernel" from "width correction". The second is the hybrid with a freely fitted exponent instead of the theoretical one. Finally, a "small-proxy" baseline picks the best configuration at $N=100$ and deploys it unchanged at large $N$. It tests the opposite error: assuming the optimum does not move.
Why not just…?
Why not just grid-search at the deployment width? For small deployment widths this is cheap, and it is a genuine competitor. Our selector must at least match it, and the experiments report rollout counts alongside scores. The cost argument applies at large widths and large grids, where the free-probability paper reports 156,600 rollouts avoided on its suite. Separately from cost, a grid search only gives an answer. It does not explain why the best spectral radius moves with width, and it does not extrapolate to widths not yet run.
Why not just use the infinite-width kernel, linear or tanh? Under our hypothesis this choice is systematically biased toward the stability edge at moderate width. The ablation without the correction term tests exactly this. If its selections match our corrected selector at every width, the drift claim is false, and the contribution reduces to the rate measurements.
Why not just fit a learning curve to a few small-width rollouts? This is the most serious threat. A free-exponent fit might work as well as theory-guided extrapolation. We run it as an ablation. The theory earns its place only if one of two things holds. Either fixing the exponent at 1 makes extrapolation from two widths markedly more stable than free fitting, or the kernel-computed $a(\theta)$ removes the need for rollouts entirely. If neither holds, we will report that empirical curve fitting suffices. The remaining contribution is then the explanation: the $1/N$-versus-$1/\sqrt{N}$ separation between average risk and spread, and the predicted breakdown near the edge.
Why not just train an RNN with backpropagation? This answers a different question. Fully trained gated networks can lose to wide reservoirs under matched trainable-parameter budgets (Micheli et al.) and overfit in low-data settings (Hinaut). We are asking how wide a reservoir must be, not whether reservoirs should be used, so we do not include trained RNNs.
Proposed Experiments and Decisive Checks
The main testbed is the ten-task synthetic suite: NARMA-10, NARMA-20 and NARMA-50; memory capacity; Lorenz63; Lorenz96; and Mackey-Glass. We score with $1-\text{NRMSE}$ for direct comparability with the published 0.774 and 0.772. Widths run from 25 to 8,000 on a logarithmic grid, with spectral radius from 0.5 to 1.1. We use at least 50 reservoir seeds per cell, which is cheap because each readout is one linear solve.
For real data we use the ETT forecasting sets. No reported result was found on Lacuna for reservoirs on ETT, so there we compare only against our own grid-search reference. As a secondary classification check, we use a subset of the UCR archive with a plain bidirectional echo state network. We do not claim to beat SARC's reported 85.08% average accuracy from Spectral-Aware Reservoir Computing for Fast and Accurate Time Series Classification, since that method changes the architecture.
Four checks decide the proposal:
- Exponents. On a log-log plot of the mean risk gap against $N$, the slope should be close to $-1$ in the contractive region, while the seed spread should have slope close to $-1/2$. Slopes near $-1/2$ for the mean would falsify the weak-error claim.
- Coefficient prediction. The kernel-computed $a(\theta)$ should predict the measured coefficient's rank order across hyperparameters. It should also predict its growth toward $\rho\approx1$, including the point where the expansion visibly fails.
- Drift of the optimum. The best spectral radius should shift monotonically with width, in the predicted direction and by the predicted amount.
- Selection quality. At matched rollout budgets, the corrected selector should close most of the gap between the uncorrected kernel selector and deployment-width grid search at small $N$. At large $N$, all three should converge.
References
- Dynamics of Finite Width Kernel and Prediction Fluctuations in Mean Field Neural Networks
- Quantitative Gaussian approximation of randomly initialized deep neural networks
- Approximation bounds for random neural networks and reservoir systems
- Infinite-dimensional reservoir computing
- Risk Bounds for Reservoir Computing
- Free-Probability Kernels for Zero-Rollout Hyperparameter Selection in Reservoir Computing
- Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes
- Non-asymptotic approximations of neural networks by Gaussian processes
- Comparison between DeepESNs and gated RNNs on multivariate time-series prediction
- Robot Language Acquisition Modelling via Cross-Situational Learning with Little Data
- Spectral-Aware Reservoir Computing for Fast and Accurate Time Series Classification