Continual learning methods are usually compared at a fixed "buffer size", meaning the number of past examples they may keep for rehearsal. The best-known brain-inspired method, which keeps a fast-changing and a slow-changing copy of the network alongside the one being trained, is credited with large gains at a 200-example buffer. Those two extra copies are themselves memory, and they are not counted. For a network of roughly eleven million parameters stored at 32 bits, the two copies take about 89 MB. A 64×64 color image stored raw takes about 12 KB, so the same 89 MB would hold roughly 7,000 images, against the 200 the method is charged for. (This is our own arithmetic; the backbone size is an assumption, not a figure from the cited papers.) The headline comparison therefore sets methods with very different physical memory against each other.
This proposal treats a complementary learning system as a data structure with a total byte budget and a cost per training step. Under that view the slow model and the example buffer compete for the same bytes: both are ways of remembering the past. We ask which kind of storage is worth more per byte. We also propose a specific way to make the slow model cheap. It is stored at very low precision and updated with randomized rounding, so that its tiny averaging steps are not rounded away to zero. The bytes this saves go to more, and lightly compressed, rehearsal examples.
Core Claim
CLS-ER trains a working network and keeps two exponential moving averages of its weights. An exponential moving average (EMA) is a running average that drifts slowly toward the current weights. One average is fast ("plastic") and one is slow ("stable"). The method rehearses examples from a small buffer and uses whichever average is more confident as a teacher. It reports 23.47% on S-Tiny-ImageNet with 200 buffered examples, against 10.96% for DER++. Memory Replay with Data Compression (MRDC) showed a related point inside the buffer alone: more JPEG-compressed examples beat fewer raw ones. Adding MRDC to LUCIR lifts 25-phase ImageNet-sub accuracy from 61.44% to 70.53%. MRDC also found a sharp quality threshold below which compressed images become out-of-distribution and hurt.
Our claim combines these two results. Under honest byte accounting, the slow and fast weight copies are a poor use of memory at full precision. They tolerate aggressive compression because they only ever supply soft targets. The best system at a given byte budget pairs a low-bit, stochastically rounded slow teacher with a much larger compressed buffer.
Concretely, we predict three things on the split-image benchmarks. First, plain rehearsal given CLS-ER's full footprint as raw images will close most of CLS-ER's published gap over DER++. Second, a 4-bit stable teacher with stochastic rounding will keep most of CLS-ER's benefit, while a deterministically rounded one will freeze and lose it. Third, the combined design will dominate both on the curve of accuracy against total bytes. If the first prediction fails, so that teachers beat examples even byte-for-byte, that is itself a useful finding: weight averages would then store knowledge more efficiently than exemplars.
The memory-speed literature explains why the split can matter. Maximum Likelihood Associative Memories shows that neural associative memories store patterns far above their information content, from 131% overhead for Gripon-Berrou networks to over 11,000% for Hopfield networks at a 1% error rate. That motivates suspicion that weights are an expensive place to keep the past. It concerns stored binary patterns, not EMA teachers, so it motivates our question rather than answering it.
Research Program
The first piece is an accounting harness. Every method is charged for the bytes of the buffer, any extra weight copies, and any stored logits. The working model is charged equally for all methods, so it is excluded. Every method is also charged for wall-clock time and floating-point operations per training step, including decoding.
This second charge matters. Succinct data structures teach that compressed storage pays a decoding tax: Orzo's Elias-Fano-compressed rank index (Elias-Fano Compression for Space-Efficient Rank and Select Structures) saves 26.49% space but slows queries by 9–38%. SPIDER shows that layout chosen around cache misses can recover speed at small overhead. MRDC never measured its JPEG decoding cost during training. CLS-ER already pays two extra forward passes per rehearsal step. We report both axes, so a method cannot win on bytes by quietly losing on time.
The second piece is the succinct slow learner. The stable copy is held in $b$-bit per-channel quantized form, with $b \in \{8, 4, 2\}$. Its update is $\theta_s \leftarrow \mathrm{SR}_b(\alpha\,\theta_s + (1-\alpha)\,\theta_w)$. Here $\theta_s$ is the stored stable weights, $\theta_w$ the working weights, $\alpha$ the averaging rate (close to 1), and $\mathrm{SR}_b$ stochastic rounding to the $b$-bit grid. Stochastic rounding is unbiased in expectation, so increments far smaller than one quantization step still accumulate on average. Deterministic rounding would discard them, and the slow copy would stop moving. The randomized-quantization idea comes from Learning Succinct Models. That work compressed a static 27 MB model to 13 KB with F1 moving only from 0.9730 to 0.9720. We move its idea from offline compression to a continuously updated copy, which that work never addressed. The fast copy gets the same treatment, with its bit-width as a separate knob. At teacher time, weights are dequantized one layer at a time, and we measure that cost.
The third piece is an allocation rule, modelled on MRDC's rule for picking the lowest JPEG quality before its feature-volume cliff. We pick the lowest teacher bit-width at which the compressed teacher still agrees with a full-precision shadow teacher on buffered examples, measured as KL divergence below a threshold on a short calibration prefix of the stream. All remaining bytes go to exemplars at MRDC's selected JPEG quality. The rule is cheap. The experiments will check whether it lands near the empirically best point on the frontier.
Baselines
The naive reference is plain experience replay (ER) with reservoir sampling. It is run at each byte budget, with the whole budget spent on raw images, and again with MRDC-selected JPEG images. If this baseline alone matches CLS-ER at CLS-ER's own footprint, then much of the published dual-teacher advantage comes from uncounted memory.
The strongest existing method is CLS-ER itself, since it holds the best number in the file on our main benchmark: 23.47% on S-Tiny-ImageNet with 200 examples (CLS-ER). We run it exactly as published at full precision, and also with its buffer replaced by an MRDC-compressed buffer of the same byte size. This gives CLS-ER the full benefit of the best known buffer compression. DER++, at 10.96% in the same setting, is included at matched bytes, since it spends memory on stored logits rather than teacher copies.
For the incremental ImageNet-sub setting, the reference is LUCIR+MRDC at 70.53% on the 25-phase protocol (MRDC). There we add our slow learner and charge it bytes. The file reports no CLS-ER number on ImageNet-sub; we found no reported result on Lacuna for CLS-ER on that benchmark, so we will run it.
All methods share one backbone, one stream order per seed, the same number of gradient steps and batch size, and five seeds. Each method gets the same tuning budget: a fixed number of hyperparameter trials, tuned on a held-out validation split of the first two tasks.
Three ablations isolate the key ingredient. The first uses deterministic rounding at the same bit-width; this tests whether stochastic rounding is what keeps a low-bit slow copy alive. The second keeps only a single EMA teacher, testing whether one of the two copies is the redundant part. The third keeps the full-precision teachers but shrinks the buffer to pay for them, which is CLS-ER's real operating point once all bytes are counted.
Beating CLS-ER means one of two things. Our design must give higher final average accuracy at equal total bytes, beyond seed variance. Or it must give equal accuracy with at least half the bytes, with time per step no worse than CLS-ER's.
Why not just…?
Why not just quantize CLS-ER's teachers to 8 bits and stop there? This is the closest threat, and it is part of our sweep rather than a separate approach. Eight-bit storage with ordinary rounding may well work, because 8-bit increments are fine enough to capture the slow average's drift. The question is what happens at 4 and 2 bits. There, our analysis predicts that deterministic rounding stalls the slow average. If 8-bit deterministic quantization already reaches the frontier, we will report that as the answer. The contribution then becomes the byte-accounting result and the reallocation of saved memory, not the rounding scheme.
Why not just drop the teachers and fill the memory with JPEG exemplars, MRDC-style? This might win. It is the first decisive check, because MRDC shows example quantity pays off strongly and no paper in the file compares teachers against exemplars byte-for-byte. But MRDC also reports a quality cliff: below a JPEG quality around 50, extra examples turn harmful. So exemplar-only scaling has a ceiling at small budgets. A cheap teacher may add value precisely past that ceiling. The byte-sweep curves will show where, if anywhere, the two lines cross.
Why not just store logits instead of teacher weights, as DER++ does? Stored logits are cheap per example, but they are frozen at write time. EMA teachers, by contrast, keep producing targets that reflect consolidated, current knowledge. CLS-ER's large published margin over DER++ is weak evidence that this matters, but it is not byte-matched. Byte-matched DER++ is therefore one of our baselines and settles the question directly.
Proposed Experiments
The main experiment sweeps total extra memory budgets. The sweep runs from below the cost of one full-precision teacher up to CLS-ER's full footprint. It covers S-CIFAR-10, S-Tiny-ImageNet, and the general continual setting GCIL-CIFAR-100, including its long-tailed variant, all as defined in CLS-ER. It also covers 25-phase ImageNet-sub as in MRDC. The file reports no CLS-ER numbers for S-CIFAR-10 or GCIL-CIFAR-100, so we found no reported result on Lacuna for those, and all such numbers will come from our own runs. Metrics are final average accuracy, average forgetting, calibration error (CLS-ER claims better calibration, and compression could erode it), and wall-clock time and floating-point operations per step, including decoding. Results are plotted as accuracy against bytes, with time per step shown alongside.
Three checks decide the proposal. The first gives plain ER and DER++ CLS-ER's full counted footprint as exemplars and compares them on S-Tiny-ImageNet. If either comes within two points of 23.47%, the field's buffer-size convention is materially misleading. The second compares 4-bit stochastic-rounded against deterministically rounded slow teachers. We track how far each stored copy drifts from a full-precision shadow average over the stream. Our mechanism predicts the deterministic copy freezes while the stochastic one tracks; if both track, the rounding argument is wrong. The third tests whether the allocation rule's chosen operating point lands within one point of the best grid point on at least three of the four benchmarks. If it does, the rule is a usable cheap recipe. If it does not, the frontier itself is the result, and the rule needs replacing.
References
- Learning Fast, Learning Slow: A General Continual Learning Method based on Complementary Learning System
- Memory Replay with Data Compression for Continual Learning
- Maximum likelihood associative memories
- Elias-Fano Compression for Space-Efficient Rank and Select Structures
- SPIDER: Improved Succinct Rank and Select Performance
- Learning Succinct Models: Pipelined Compression with L1-Regularization, Hashing, Elias-Fano Indices, and Quantization