Replay Buffers That Never Go Stale: Choosing What to Remember in the Fixed State Space of a 2D Reservoir

A model that learns from a stream of tasks forgets old tasks as it trains on new ones. The standard fix is experience replay: keep a small memory buffer of past examples and mix them into training. When the buffer is small, which examples you keep matters a great deal. The best selection rules pick examples that are spread out in the model's feature space. Those rules have a built-in flaw, though. They measure "spread out" with the same features the network is busy retraining, so the distances used to choose the buffer stop being valid as soon as the network updates.

We propose removing the flaw at its source. The images are passed through a reservoir, a large recurrent network whose weights are random and never trained. Only a linear output layer is fitted, in closed form. Because the features never move, an example's value to the buffer can be scored once, when it arrives, and that score stays correct for the rest of the stream. With fixed features and a linear readout, "keep the most spread-out examples" also stops being a heuristic. It becomes a classical optimal-design criterion: keep the examples that best pin down the readout weights. The remaining obstacle is that existing vision reservoirs see an image as a one-dimensional sequence of patches, which gives weak spatial context. We therefore build a two-dimensional reservoir that spreads information across the whole image grid without any training. The resulting question is sharp and testable: at a fixed memory budget in bytes, does diversity-selected replay in a drift-free reservoir space beat both random replay and the obvious reservoir alternative of storing running covariance statistics?

Core Claim

Three claims follow, and each can be tested.

First, diversity selection helps more in a fixed feature space than in a drifting one. In a drifting space the selection scores go stale. In a fixed space they do not, so the same log-determinant selection rule should gain more over random selection when features are fixed.

Second, at memory budgets typical of replay benchmarks, a diverse buffer of raw images beats storing exact sufficient statistics for the readout. A raw image costs a few kilobytes, while the covariance matrix a ridge readout needs grows with the square of the reservoir width. Buffered images can be re-encoded for free because the reservoir never changes.

Third, the diversity metric is only as good as the geometry of the reservoir states. A reservoir with whole-image spatial context should give more useful selections than one with only local random filters. It should also do better than the one-directional patch scan of ViR: the Vision Reservoir.

Position Relative to Prior Work

Several papers motivate the gap by showing that diversity-aware selection pays off, always in trained or drifting feature spaces.

  • READMem admits a video frame into memory only if it enlarges the determinant of the Gram matrix of stored embeddings, a measure of the volume they span. This raised long-video segmentation from 64.3 to 86.0 J&F, a combined overlap-and-contour score, on the LV1 long-video benchmark. It works at inference time on a pretrained encoder and never updates that encoder.
  • CoVIO prunes the most cosine-redundant exemplar and roughly halves translation error on KITTI. Its encoder keeps training, so its similarities drift.
  • ESMER documents "abrupt representation drift" under replay and counters it with a second, slowly updated copy of the network plus loss-aware buffering. Its "reservoir sampling" is the classic statistical streaming algorithm, not reservoir computing.

We keep the volume-based selection of READMem. We move it into a space that provably cannot drift, and we use it during continual training rather than only at inference.

Reservoir computing supplies the method. A reservoir is kept stable by scaling its recurrent weights to spectral radius below one, the "echo state property" (Gallicchio et al., Gonon et al.). Its readout is trained by ridge regression (Jaeger). Reservoir states can be cached and reused (Lukosevicius et al.).

The closest continual-learning work, Competitive Federated Reservoir Computing, shows that fixed reservoirs do not suffer feature forgetting. It avoids replay entirely by accumulating the ridge-regression statistics for each readout head. That paper both motivates and constrains us: it is exactly the "no buffer needed" competitor we must beat. Its experiments are low-dimensional dynamical systems, where storing a covariance matrix is cheap. In vision the reservoir must be thousands of units wide, and this trade-off flips.

For spatial context, several papers show how to reach the whole image cheaply, but always with trained weights.

  • Multi-Dimensional Recurrent Neural Networks sweep trained recurrent units across the image grid in linear time.
  • CCNet reaches every pixel by applying row-and-column attention twice, at O(N√N) cost, where N is the number of pixels.
  • ViR is the only fixed-reservoir vision model in the file. It reports 51.98% on CIFAR-100 from scratch against 45.61% for a ViT of similar depth. It flattens the image into one scan order and was never tested in continual learning.

We borrow CCNet's two-step routing pattern but implement it with fixed random recurrences.

Research Program

The criss-cross reservoir. Each image is split into a grid of patches. Every patch is lifted by a fixed random input projection. A fixed random recurrent map is then run along each row, in both directions and in parallel across rows. A second fixed recurrent map is run along each column of the row states, also bidirectionally and in parallel. After these two passes, every patch's state depends on every other patch, the same reachability that recurrent criss-cross attention achieves. The sequential depth is only H + W steps for an H × W patch grid, which avoids the strict wavefront ordering of classic multi-dimensional recurrence.

Weights are scaled so each one-directional recurrence satisfies the echo state property. Pooled row, column and local states are concatenated into a feature vector of width D, between 2k and 8k. A four-corner sweep in the style of Graves et al., also with fixed weights, is built as a variant. Nothing in the encoder is trained, and hyperparameters such as spectral radius and input scale are set once on a held-out first task.

Readout and replay. Let $S$ be the matrix whose rows are reservoir states and $Y$ the one-hot labels. The readout is the ridge solution

$$W = (S^\top S + \lambda I)^{-1} S^\top Y,$$

refitted at each task boundary on current-task states plus buffer states. Because the reservoir is fixed, the buffer can store raw images; their states are recomputed exactly and cheaply.

Buffer selection. The buffer $\mathcal{B}$ is chosen to maximize the log-determinant of its regularized Gram matrix,

$$\mathcal{B}^\star = \arg\max_{|\mathcal{B}| = B}\ \log\det\left(\lambda I + S_{\mathcal{B}}^\top S_{\mathcal{B}}\right),$$

where $B$ is the buffer size and $S_{\mathcal{B}}$ holds the states of buffered examples. For a linear readout on fixed features, the uncertainty in the readout weights is proportional to $(\lambda I + S^\top S)^{-1}$. Maximizing this determinant therefore shrinks the volume of that uncertainty; this is the D-optimal design criterion, which READMem's Gramian rule resembles only informally.

The method runs online. An incoming example's gain is its leverage, $\log(1 + s^\top(\lambda I + S_{\mathcal{B}}^\top S_{\mathcal{B}})^{-1}s)$, computed through the $B \times B$ kernel so the cost is $O(B^2)$ per candidate rather than cubic in D. When the buffer is full, the stored example with the lowest leave-one-out leverage is evicted. Per-class quotas prevent one class from monopolizing memory.

Pure volume maximization tends to hoard outliers and mislabeled points. To counter this, a variant scores leverage within each class and rejects candidates that the current readout finds grossly inconsistent with their label. This borrows ESMER's insight that low-loss anchors are safer.

Baselines

The naive reference point is a reservoir readout refit on the current task only, with no memory. It shows how much forgetting a fixed-feature model suffers when the readout alone is overwritten.

The standard replay reference is the same reservoir and readout with a buffer filled by uniform streaming random sampling. It uses the identical byte budget, so any gain from our method is attributable to selection alone.

The strongest existing method in the file is ESMER on its own benchmarks, a fully trained network with a slow-updating stable copy and error-sensitive buffering. ESMER reports 48.77% final accuracy on Seq-CIFAR-100 with buffer 200, against 43.80% for CLS-ER and 21.91% for plain experience replay. It reports 36.83% on Seq-CIFAR-10 with buffer 10, against 22.20% for plain replay. We rerun ESMER from its published setting on our splits with its own tuning, count its buffer in bytes, and report training time alongside accuracy.

Beating ESMER outright with an untrained encoder is not the bar we set. Two outcomes would count as a clear win. The first is matching or exceeding ESMER at buffer 10 on Seq-CIFAR-10, the regime where sample choice matters most. The second is exceeding CLS-ER on Seq-CIFAR-100 at buffer 200 while training in minutes rather than hours.

The strongest reservoir-native competitor is covariance accumulation in the style of Bereska et al.. The readout's sufficient statistics $S^\top S$ and $S^\top Y$ are accumulated over the whole stream, which reproduces joint training exactly. Its reservoir width is shrunk until the statistics fit the same byte budget, and the comparison is swept across budgets and widths. Beating it means a higher final accuracy at equal bytes. Finding the budget where the curves cross is itself a result.

Four ablations isolate our ingredients.

  • Staleness. The same log-determinant selection runs on a small backpropagated convolutional encoder whose scores are computed at admission and never refreshed. This isolates whether the fixed space is what makes selection work.
  • Selection rule. Log-determinant selection is swapped for CoVIO-style cosine pruning.
  • Spatial context. The criss-cross reservoir is replaced by a ViR-style one-dimensional scan, and separately by a fixed random convolutional encoder with only local receptive fields, all at equal width D.
  • Noise robustness. The class-conditional, loss-filtered variant is compared with plain volume selection on noisy labels.

Why not just…?

Why not just accumulate the covariance and skip the buffer? For a linear readout on fixed features this gives exact joint training, and in low dimensions it is clearly the right answer; that is why Bereska et al. use it. Its memory, however, grows as D². At D = 4096 in 32-bit floats it needs about 64 MB, whereas 200 CIFAR images take about 600 KB. At replay-sized budgets the covariance route forces a much narrower reservoir. This is a real threat, not a strawman. If narrow-reservoir covariance matches wide-reservoir replay at the budgets in ESMER's tables, the buffer half of this proposal fails, and the byte-matched sweep is designed to show that directly.

Why not just freeze a pretrained ResNet or CLIP encoder instead of a random reservoir? It is equally drift-free and its features are much stronger, so it will likely win on CIFAR in absolute accuracy. It answers a different question, though, because it imports large external data and tens of millions of parameters. ViR is reported at 6.24M parameters against 41.66M for its ViT counterpart. We run the frozen-backbone variant with the identical selection rule and report it separately. If the selection gain (log-determinant over random) is similar in both fixed spaces, that supports claim one independently of the encoder, and the reservoir becomes the choice for settings without pretraining.

Why not just use random replay, since the features are fixed anyway? Fixed features remove drift, not redundancy. CoVIO reports that small diversity-selected buffers beat larger random ones. The optimal-design argument predicts the gap should be largest at the smallest budgets, so buffer 10 is the deciding case. If random sampling matches log-determinant selection there, claim one is false.

Why not just use ViR as published, or random local convolutions, for the reservoir? Both are ablations here. If either matches the criss-cross reservoir on selection gain and final accuracy, the spatial component adds nothing and we drop it.

Proposed Experiments

The main benchmarks are Seq-CIFAR-10 and Seq-CIFAR-100, each class-incremental over five tasks, with buffers of 10, 200 and 500 examples measured in bytes. We also run the 50%-label-noise Seq-CIFAR-10 variant, on which ESMER reports a 116% gain over plain replay. Metrics are final average accuracy, average forgetting, wall-clock training time, and memory in bytes. The upper bound is a ridge readout trained jointly on the full reservoir features of all tasks, i.e. the joint i.i.d. solution.

Three checks decide the proposal. The first tests staleness: the log-determinant gain over random must be significantly larger in the reservoir space than in the drifting encoder, across five seeds. The second is the byte-matched crossover curve against covariance accumulation. The third is the spatial ablation: the criss-cross reservoir must beat both the one-directional scan and local random features on selection gain, not merely on raw accuracy. It would be a small but useful spatial-selection result if class-conditional leverage selected fewer mislabeled examples than plain volume selection on the noisy split.

If the first two checks succeed, a natural second application is streaming odometry on KITTI, where CoVIO's diversity buffer suffers exactly the drift this design removes. We present it only as a follow-up.