A reservoir computer is a recurrent network whose recurrent weights are fixed and never trained; only a final linear layer (the "readout") is fit, usually by ridge regression. When the fixed recurrent part is a ring, each neuron passes its value to the next neuron in a closed loop. Prior work has shown that such a ring needs no stored weights and no multiplications, since one step is just a cyclic shift. Running a ring reservoir on a small chip is therefore cheap. Training it on the chip is not. Ridge regression accumulates an $N \times N$ matrix of state correlations, which for a 1000-unit reservoir in 32-bit floats takes about 4 MB. That is far beyond the SRAM of a microcontroller-class device. Today the ring's hardware advantage ends at inference, and the readout is fit offline.
This note proposes making the training step as cheap as the inference step. The ring's shift structure causes the state-correlation matrix to be approximately circulant, meaning each row is a cyclic shift of the one before. A circulant matrix is diagonalized by the Fourier transform, so it can be summarized by $N$ numbers and inverted in $O(N \log N)$ time. We use this summary to precondition a streaming conjugate-gradient solver, which recomputes reservoir states on each pass over the data. The solver's working memory is linear in reservoir size. The goal is a readout that matches exact ridge regression, trained in a few passes, with peak training memory small enough for on-chip SRAM. This would let the memory ring scale to thousands of units on devices where today only about 100 units are practical.
Core Claim
Take a linear ring memory $m(t) = P\,m(t-1) + V u(t)$. Here $m(t) \in \mathbb{R}^N$ is the memory state, $P$ is the cyclic shift matrix, $u(t) \in \mathbb{R}^d$ is the input, and $V \in \{\pm 1\}^{N \times d}$ holds deterministic input signs. The Gram matrix $G = \sum_t m(t) m(t)^\top$ then has a nearly circulant structure. To see why, consider a scalar stationary input. The expected correlation between units $i$ and $j$ is a sum over delays of $v_{i-k} v_{j-k}$, which depends only on $(i-j) \bmod N$ once the sum covers full wraps of the ring. The best circulant approximation of $G$ has eigenvalues
$$ s = \sum_t \left|\mathcal{F} m(t)\right|^2 , $$
where $\mathcal{F}$ is the length-$N$ discrete Fourier transform. These can be accumulated in one streaming pass using $O(N)$ memory. We then solve the ridge system $(G + \lambda I) w = b$ for each output column with conjugate gradient. Here $\lambda$ is the ridge penalty and $b = \sum_t m(t) y(t)$ is the input–target cross term. The preconditioner is $\mathcal{F}^{-1}\mathrm{diag}(s+\lambda)^{-1}\mathcal{F}$. Each product $Gp$ is computed by regenerating states on a pass over the training stream, so $G$ is never stored.
We make two claims. First, this reaches the exact ridge solution's accuracy in a small number of passes, roughly 5 to 15, while unpreconditioned gradient methods need far more. Ring states are strongly correlated, which makes the problem badly conditioned, and the circulant preconditioner targets exactly that correlation. Second, at unit spectral radius with integer input signs, the ring itself can be implemented exactly in integers. In that case $P$ is a permutation, so the update adds no rounding error and cannot drift. This answers concretely the open worry about quantizing marginally stable memory rings.
Research Program
The mechanism comes from three lines of work. Input-to-State Representation in Linear Reservoirs Dynamics shows that a 1000-unit cyclic reservoir has a full-rank controllability matrix, meaning its state can encode every input from the last $N$ steps. It reports rank about 274 for random reservoirs and about 57 for Wigner reservoirs. This motivates using rings as the memory. It also warns of the cost: full-rank, slowly decaying states yield a Gram matrix that is both large and ill-conditioned.
Reservoir Memory Networks and Residual Reservoir Memory Networks supply the architecture we adopt. They split the reservoir into a linear ring memory driven by the input and a small nonlinear reservoir that reads from it, at spectral radius 1. Both were simulated only in floating point, with nonlinear modules of 100 or fewer units. Efficient Recurrent Neural Networks using Structured Matrices in FPGAs supplies the tool: storing circulant blocks as one row and multiplying them via FFT. That paper imposes circulant structure on trained LSTM weights. We instead exploit circulant structure that appears on its own in the statistics of an untrained ring, and we use it for training, not inference.
The build has three parts. The first is an inference kernel. The ring memory is a pointer into a static SRAM circular buffer, so no data moves between steps. Each step adds $V u(t)$ at the rotated offset and advances the pointer, with integer accumulators sized to $\lceil \log_2(T \max|u|) \rceil$ bits, where $T$ is the sequence length. Input and inter-module signs are regenerated from a deterministic digit sequence, as in Simplifying Deep Reservoir Architectures. The nonlinear module is a small ring with clipping, as in Integer Echo State Networks: Efficient Reservoir Computing for Digital Hardware.
The second part is the streaming solver. It handles the joint readout over the memory state and the nonlinear state with a block preconditioner: circulant for the large memory block, and a dense inverse for the small nonlinear block, at most 100 by 100. Peak memory is about $O(N_m + C(N_m + N_{nl}) + N_{nl}^2)$, where $N_m$ is the memory size, $N_{nl}$ the nonlinear module size, and $C$ the number of outputs.
The third part is analysis. We measure how far the true $G$ is from its circulant projection, $\|G - \mathrm{circ}(G)\|/\|G\|$, as a function of decay factor, input color, number of input channels, and sequence length relative to $N$. We also relate this distance to the observed CG iteration count. The standard result that preconditioned CG converges at a rate set by the spread of the preconditioned spectrum gives a testable prediction: pass counts should grow as the circulant mismatch grows.
Baselines
The upper reference is exact dense ridge regression on the same reservoir, computed offline in float64 with $\lambda$ tuned by validation. It defines the accuracy target. Our method "beats" this baseline if it comes within 0.5 accuracy points on classification, or within 5% relative MSE on regression, while using a small fraction of its training memory.
The do-nothing on-chip reference is dense ridge on the largest reservoir whose Gram matrix fits the same SRAM budget, for example roughly 200 total units in 160 KB. This is what a practitioner would do today. Our method beats it if the larger ring trained with $O(N)$ memory is more accurate at equal peak bytes.
The strongest training competitor is plain streaming gradient descent or Adam on the readout. Its memory is also $O(CN)$, so it attacks the same niche directly. It gets the same number of data passes as our solver, with its learning rate tuned over a grid of equal size to our $\lambda$ grid. Diagonal (Jacobi) preconditioned CG serves as the middle ablation. It isolates whether the circulant structure, rather than preconditioning in general, is what matters. Recursive least squares is included as an exact online method, but at $O(N^2)$ memory it is mainly a reference point.
The strongest prior systems on accuracy are the reservoir memory networks themselves. Residual Reservoir Memory Networks reports a 19.9% average accuracy improvement over LeakyESN across 12 UEA/UCR tasks. Reservoir Memory Networks reports 95% on sequential MNIST for its Legendre-memory variant, versus about 77% for a standard ESN. We rerun the ring variants at their published sizes in float with exact ridge to confirm we reproduce them. We then run them at larger memory sizes trained on-chip.
For the trained-network alternative under an on-chip budget, Dynamic Hand Gesture Recognition for Wearable Devices with Low Complexity Recurrent Neural Networks reports 11.43% error on the SmartWatch accelerometer gesture task in 17.25 KB, using a 2-bit LSTM. We match that byte budget. For intESN, the file reports only accuracy "comparable" to float ESNs on its UCR/UCI datasets, so no reported result was found on Lacuna for its per-dataset accuracy. We rerun it ourselves.
The key ingredient ablation is a raw-input delay buffer of equal bytes, which stores the last $W$ inputs, with the same streaming solver. Its Gram matrix is Toeplitz, which is also nearly circulant. This separates the benefit of the ring's folded memory over all $T$ steps from the benefit of the solver.
Why not just…?
Why not just train the readout with SGD, which already needs only $O(CN)$ memory? This is the most serious threat. If tuned Adam reaches exact-ridge accuracy in as many passes as preconditioned CG, the solver contribution collapses to "use SGD." Our expectation that it does not rests on the full-rank, slowly decaying ring states documented in Input-to-State Representation in Linear Reservoirs Dynamics. These states produce a correlated, poorly conditioned Gram matrix, and first-order methods are known to struggle on such problems. The pass-count curves against SGD, Adam, and Jacobi-CG settle the question directly. We will report honestly if the gap is small on short sequences.
Why not just use a smaller reservoir whose Gram matrix fits? Accuracy of ring-memory networks depends on memory length, and the published results run at 100 or fewer nonlinear units with unexplored scaling. We will plot accuracy against reservoir size at fixed SRAM. If the curve is already flat at the size dense ridge can afford, the proposal has no use case. We will say so.
Why not just buffer the raw input window instead of running a ring? For a single input channel and $T \le N$, a full-rank linear ring is an invertible transform of the raw window. A linear readout on it is then equivalent to one on the raw buffer, so in that regime the ring adds nothing. The ring can only win when $dT > N$. Then it keeps an aliased summary of the whole sequence where a buffer drops older steps. It also helps when the nonlinear module needs mixed memory as input. The equal-byte delay-buffer ablation on sequential MNIST (784 steps) and the multichannel UEA tasks tests exactly this case.
Why not just quantize a trained LSTM? At the 17.25 KB budget, the 2-bit LSTM in Dynamic Hand Gesture Recognition for Wearable Devices with Low Complexity Recurrent Neural Networks is strong, but it requires backpropagation-through-time retraining off-device. Our comparison is at equal inference bytes on SmartWatch gestures. We additionally report training memory, where only the reservoir can be trained on the device at all.
Proposed Experiments
We will run on the 12 UEA/UCR tasks used by ResRMN, sequential and permuted sequential MNIST, SmartWatch gestures, and Mackey-Glass, NARMA10, and Santa Fe Laser for regression. On the regression tasks, Simplifying Deep Reservoir Architectures reports Laser MSE of $1.15 \times 10^{-3}$ for a deterministic ring versus $3.09 \times 10^{-3}$ for a sparse reservoir. We sweep memory sizes from 100 to 4096 units. For each run we measure accuracy or MSE, peak training SRAM, number of data passes, and wall-clock time on a Cortex-M-class microcontroller. We also measure inference cycles and energy per sample for the pointer-buffer kernel against an explicit-shift implementation.
The first decisive check is the circulant-mismatch study. If $\|G - \mathrm{circ}(G)\|/\|G\|$ is large on real data and CG pass counts track it upward, the preconditioner fails where it matters. The second check is the SGD comparison at equal passes. The third check is whether a large on-chip-trained ring beats both the largest dense-ridge reservoir that fits in SRAM and the equal-byte delay buffer, on the long multichannel tasks. If all three checks favor the proposal, ring reservoirs become the first recurrent models that are both run and trained entirely within a microcontroller's SRAM.
References
- Input-to-State Representation in Linear Reservoirs Dynamics
- Reservoir Memory Networks
- Residual Reservoir Memory Networks
- Efficient Recurrent Neural Networks using Structured Matrices in FPGAs
- Simplifying Deep Reservoir Architectures
- Integer Echo State Networks: Efficient Reservoir Computing for Digital Hardware
- Dynamic hand gesture recognition for wearable devices with low complexity recurrent neural networks