Slow Reservoirs: Learning Recurrent Weights at a Curvature-Matched Rate

Recurrent networks are usually trained in one of two extreme ways. In the first, every weight is updated with backpropagation through time (BPTT), meaning gradients are passed backward through the whole unrolled sequence, all with the same learning rate. In the second, called reservoir computing, the recurrent weights are drawn at random, frozen, and only a final output layer (the "readout") is trained. Seen through the lens of per-layer learning rates, these are the two endpoints of a single dial: the recurrent weights' learning rate is either equal to the readout's or exactly zero. This note proposes to study the interior of that dial. We start from a well-built reservoir, then let its recurrent weights move at a small rate that is set from measured curvature and updated during training. We test whether this gets the fast start and stability of reservoirs together with the higher final accuracy of fully trained networks.

The motivation is an unresolved tension between two recent results. When Learning Hurts: Fixed-Pole RNN for Real-Time Online Training models a linear recurrent network as a bank of filters whose "poles" are set by the recurrent weights. It shows that learning those poles produces a badly conditioned loss surface: the curvature condition number is about $10^3$ for a trainable network against about 2.5 for a fixed-pole echo state network, and the poles drift beyond the stable region. That paper, however, only compares the two endpoints. Balancing Learning Rates Across Layers proves that in linear networks the best first step uses unequal layer rates, while by the second step the layers must coordinate multiplicatively. That result suggests that freezing one layer entirely gives up real progress. Our claim is that the ill-conditioning in the first paper comes from poles moving far and clustering together, and that a curvature-matched rate small enough to keep poles near their reservoir positions can capture the coordination gain the second paper predicts without entering the bad valleys.

Core Claim

We call a "slow reservoir" an echo state network whose recurrent weights, input weights and readout are all trained. It has two defining features. First, it is initialized from a reservoir design with good memory properties: either the edge-of-stability construction of Edge of Stability Echo State Network or a stacked Deep Echo State Network (DeepESN) as surveyed in Deep Echo State Network (DeepESN): A Brief Survey. Second, it uses a recurrent learning rate tied to the readout's by a curvature ratio:

$$\eta_{\text{rec}}(t) = c \,\eta_{\text{out}}\,\frac{\lambda_{\max}(H_{\text{out}}(t))}{\lambda_{\max}(H_{\text{rec}}(t))}.$$

Here $\eta_{\text{out}}$ is the readout learning rate, $H_{\text{out}}$ and $H_{\text{rec}}$ are the loss curvature (Hessian) blocks for readout and recurrent weights, $\lambda_{\max}$ is the largest eigenvalue, estimated every few hundred steps by power iteration on Hessian-vector products, and $c \le 1$ is a single safety constant.

The rule follows the logic of The Sharpness Disparity Principle in Transformers, which argues that a uniform rate is capped by the sharpest block. We invert that logic. Each block moves at a rate appropriate to its own sharpness, so the recurrent block is automatically slowed whenever poles begin to cluster and its curvature spikes.

The falsifiable prediction has three parts. At a fixed training budget, the slow reservoir should beat the frozen reservoir in final accuracy. It should beat uniform-rate BPTT in early data efficiency. And it should match a grid-searched fixed ratio without needing that search.

Research Program

The work has a theoretical part and an empirical part.

Theory. We extend the exact two-step analysis of Pang et al. to the partially frozen case. Their two-layer linear setting is modified so that layer one starts at a random reservoir-like matrix and has rate $\epsilon\,\eta_2$ with $\epsilon$ small. We then compute how the loss after two steps depends on $\epsilon$. We expect a loss decrease that is linear in $\epsilon$ and comes from the multiplicative coordination term, plus a quadratic penalty governed by the sharpness of layer one. That combination would give an interior optimum for $\epsilon$ and a closed-form estimate of the $c$ in the rule above.

We also connect this to Morgan et al.'s filter-bank picture. There we will check numerically whether the optimal $\epsilon$ keeps the minimum distance between poles bounded away from zero, which is the condition under which their condition number stays finite.

Empirics. We map the full curve of final and early performance against the ratio $\eta_{\text{rec}}/\eta_{\text{out}}$, on a log grid from 0 to 1, on each benchmark. We track three quantities along each run:

  • the recurrent-block condition number,
  • the spectral radius of the recurrent matrix, and
  • the minimum pole separation.

A secondary question is robustness. Reservoir performance is notoriously sensitive to the chosen spectral radius and input scaling. We will test whether slow recurrent learning flattens that sensitivity, by sweeping initial spectral radius from 0.5 to 1.1 under each regime.

Baselines

Frozen reservoir (the naive reference). The recurrent weights are frozen, and the readout is fit in closed form by ridge regression or, in the online setting, by the same first-order optimizer as the proposal with $\eta_{\text{rec}}=0$. This is the $c=0$ endpoint of our dial. It is strong on memory tasks: Edge of Stability Echo State Network reports a memory capacity of 98.43±0.11 for a 100-neuron reservoir, against a maximum of 100. It also reports a mean normalized root-mean-square error of 0.05±0.11 over 50,000 autonomous steps on eight superimposed oscillators. The ridge penalty is tuned on validation data with the same trial count as every other method. On memory capacity there is little headroom, so the goal there is only to not degrade. The contests that matter are on the tasks listed below.

Uniform-rate BPTT from the same initialization, and gated RNNs. This is the $c$ equals one endpoint, trained with Adam at a single tuned rate. We also train a GRU and an LSTM of matched parameter count, tuned with the same number of trials.

This is the strongest competitor on final accuracy. Comparison between DeepESNs and gated RNNs reports that the GRU beats a 6,000-unit DeepESN on Nottingham. The file gives no numerical accuracy for this, so we report our own reproduced numbers. Beating this baseline means the slow reservoir matches or exceeds GRU accuracy on Nottingham while keeping the DeepESN's lead on Piano-midi.de, MuseData and JSBchorales.

Freeze-then-finetune. The readout is trained first by ridge regression. Everything is then unfrozen and trained at one uniform rate. This is the most plausible simpler threat and gets equal wall-clock time.

Grid-searched two-rate baseline. A fixed ratio $\eta_{\text{rec}}/\eta_{\text{out}}$ is chosen by exhaustive search. This is the ablation that isolates the curvature-matching rule. If the grid wins clearly, the rule is not the contribution, and the finding reduces to "a small fixed ratio helps."

A second ablation keeps the curvature rule but starts from a generic RNN initialization instead of a reservoir design. This tests whether the good initialization is doing the work.

Why not just…?

Why not just use Adam, which already scales steps per parameter? Adam normalizes gradient magnitudes coordinate by coordinate. The failure Morgan et al. describe, however, is a narrow valley inside the recurrent block caused by clustered poles. Rescaling individual coordinates does not straighten that valley. The layer-wise rates of One LR Doesn't Fit All likewise helped on top of AdamW in large language models, which suggests Adam alone leaves block imbalance unresolved. Still, this is a live threat for small recurrent networks. The uniform-Adam baseline, run from the identical reservoir initialization, settles it directly: if it matches the slow reservoir, our rule is unnecessary.

Why not just freeze, fit the readout, then fine-tune everything? This two-phase approach captures the fast start. Once unfrozen at a uniform rate, though, the poles are free to cluster, and the recurrent condition number should climb toward the trainable-RNN regime. We expect to see this as a late plateau or as spectral-radius drift above one in the tracked diagnostics. If freeze-then-finetune matches our method on both early efficiency and final accuracy, we will report that the simple recipe suffices.

Why not just use a larger reservoir or a deeper readout? Scaling the frozen reservoir, or adding a trained multi-layer readout as in Bidirectional deep-readout echo state networks, leaves the temporal filters fixed. That approach still trailed the GRU on the clinical dataset: F1 0.761 against 0.779. We will include a frozen DeepESN at twice our unit count with a two-layer readout. If it closes the gap on Nottingham, adapting the recurrent weights is not needed.

Proposed Experiments

The benchmarks cover four regimes.

Memory and generation tasks. These are the memory capacity task, the memory–nonlinearity trade-off task, and eight superimposed oscillators. The check is that slow learning does not erode the reservoir's memory, measured as capacity within one standard deviation of 98.43.

Nonlinear and chaotic forecasting. These are NARMA, the Santa Fe laser series and Mackey-Glass, reported as normalized root-mean-square error. The file gives no reported numbers for these tasks.

Polyphonic music. These are the four datasets from Gallicchio and Micheli's comparison, reported as frame-level accuracy.

Online and early-efficiency settings. We simulate Morgan et al.'s wireless symbol detection under drifting channels, reporting mean squared error and bit error rate. We also use the ten-task algorithmic benchmark of Benchmarking Learning Efficiency in Deep Reservoir Computing, scored by its data-efficiency metric (Weighted Average Data Efficiency). On that benchmark the reported target is that LSTMs needed ten times more steps than reservoirs on question answering.

Decisive checks. The proposal succeeds if three things hold together:

  • the ratio sweep shows an interior optimum on most tasks, strictly better than both endpoints;
  • the curvature rule lands within one validation standard error of the grid-searched ratio;
  • at the chosen rate, the recurrent condition number stays within an order of magnitude of the frozen reservoir's, and the spectral radius stays below one.

The proposal fails if either endpoint dominates across tasks. It also fails if gains appear only together with pole clustering, which would mean the benefit comes from something other than the mechanism we claim.