Boundary-Free Quantized Gradient Projection for Blurry Data Streams

Continual learning methods that protect old knowledge by "freezing directions" work like this: after a task, they record which directions in each layer's input space the old task used, then forbid later updates from moving the weights along those directions. This approach, called gradient projection memory, needs two things that real data streams do not provide. It needs a signal saying when a task has ended, and it needs a lot of memory to store the recorded directions. A recent method, Quantized Gradient Projection for Memory-Efficient Continual Learning (QGPM), solved the memory problem with outlier-aware quantization, meaning a low-bit encoding that keeps the few extreme values at full precision and compresses the rest. On ViT-B/16 it reports shrinking the stored directions from 515 MB to about 45 MB at 4 bits with under 1% accuracy loss. QGPM still finalizes, quantizes, and stores its directions only when it is told a task has ended.

We propose to remove that trigger. The method grows a quantized direction memory continuously during an online stream where classes overlap and recur and no task boundaries are announced. The central design rule is that a stored direction is quantized exactly once and never re-encoded. Every new direction is computed against the dequantized memory, so the projection the optimizer applies always matches what is actually stored, and quantization error cannot build up over thousands of small updates. The question that makes this worth doing is a byte-for-byte one. In blurry streams, simple replay of stored examples is the method to beat. We ask whether a few megabytes of 4-bit directions, used alongside a replay buffer, buy more accuracy than spending the same bytes on more stored images.

Core Claim

Making gradient projection boundary-free with quantized storage looks easy but is not. The obvious version keeps a running sketch of recent layer inputs, merges it into the stored basis every few steps, and re-quantizes the result. That version should fail slowly. Each merge rotates the basis, so the coordinates holding the extreme values move. The outlier set and scale that CINF (QGPM's "centered inlier NormalFloat" encoding) chose last time become stale. Re-quantizing then adds fresh rounding error to directions that were already correct. This mirrors the "quantization space shift" that Overcoming Forgetting Catastrophe in Quantization-Aware Training proves drives forgetting when quantization grids are re-fit on sequential data. Their setting concerned weight grids, while ours concerns a stored basis, but the mechanism is the same: grids that adapt to new data stop serving old data.

Our claim is that an append-only quantized memory avoids this drift without needing boundaries. It works as follows. New directions are extracted from a sliding sketch of the recent stream after projecting out the dequantized stored basis. Each new direction gets its own outlier set and codebook scale. A direction is added only after the energy outside the stored subspace has persisted over a window. Under these conditions, the error of the stored subspace stays fixed after each addition instead of growing with stream length. The persistence test separates two cases: lasting new structure, which should be protected, and the brief appearance of a minor "future" class in the blurry stream, which should not consume protected capacity.

Concretely, let $\hat B$ be the dequantized stored basis for a layer, and let $S_t$ be the sketch of the last $w$ minibatches of that layer's inputs. The residual energy fraction is

$$ r_t = \frac{\| S_t - \hat B \hat B^\top S_t \|_F^2}{\| S_t \|_F^2}. $$

When $r_t$ stays above a threshold $\epsilon$ for $p$ consecutive checks, the leading left singular vectors of the residual are orthonormalized against $\hat B$. They are then CINF-quantized and appended until the protected energy reaches the target. Gradients are projected as $g \leftarrow g - \lambda \hat B \hat B^\top g$. Here $\lambda \le 1$ is the relaxation applied per direction, set from each direction's measured quantization error, following QGPM's quantization-error-aware projection. Because projection uses $\hat B$ and not the unquantized vectors, the constraint is self-consistent by construction.

Research Program

The work has three parts.

The first part is a measurement study of how outliers in projection bases behave without boundaries. On i-Blurry CIFAR-100, we run full-precision streaming projection as a shadow copy of the stored memory. We track three quantities over time: the overlap between successive outlier-coordinate sets, the principal angle between the shadow subspace and the stored quantized subspace, and how these depend on merge frequency. This tests the premise directly. If outlier coordinates in bases turn out to be stable under rotation, re-quantize-on-merge would be harmless, and the append-only rule would be unnecessary complexity.

The second part builds the method itself. It has four components: a sliding sketch, the persistence-gated append rule, per-direction CINF encoding, and error-scaled relaxation. When the byte budget fills, a rare compaction step re-encodes everything once. We count and report every compaction, since each one is a controlled re-quantization event.

The third part is the equal-byte study. It places the method inside an online replay learner and trades basis bytes against replay-buffer bytes along a fixed total budget.

Prior work plays three roles here. QGPM supplies the encoding and the relaxation idea, and the claim we test is that it can be made boundary-free. Restricted Orthogonal Gradient Projection for Continual Learning (ROGO) motivates relaxation: it shows that strict orthogonality hurts forward transfer, with 63.81% versus 61.63% forward transfer over GPM on miniImageNet. However, its search for relaxable directions runs at task endpoints, which we do not have. The blurry-stream papers define the regime and the competitors.

The conflict between When Less is More: 8-bit Quantization Improves Continual Learning in Large Language Models and LifeQuant is relevant in one specific way. The first paper finds that fixed quantization grids act as a regularizer and reduce forgetting. LifeQuant finds that grids re-fit during training cause forgetting. Our append-only rule is a bet on the first finding applied to stored bases. We freeze each direction's grid rather than letting it adapt.

Baselines

The naive reference point is online experience replay with reservoir sampling, run under the i-Blurry protocol of Online Continual Learning on Class Incremental Blurry Task Configuration with Anytime Inference. That paper reports this baseline roughly 13 points below CLIB on CIFAR-10 in anytime accuracy ($A_{AUC}$, the area under the accuracy curve sampled throughout the stream).

The strongest existing method is CLIB from the same paper. It manages memory by sample-wise importance, trains on memory only, and adapts the learning rate with a t-test. It reports 46.67% $A_{AUC}$ on CIFAR-100 i-Blurry and 70.26% on CIFAR-10. It is the strongest competitor because it was designed for exactly this stream, and in this regime memory spent on examples is hard to beat. On BlurryQ we also compare against DECO from Dual-Enhanced Coreset Selection with Class-Wise Collaboration for Online Blurry Class Incremental Learning, which reports 60.03% final average accuracy on CIFAR-10.

For fairness, we will rerun CLIB and DECO ourselves on our backbone, seeds, and stream orderings. We will give each method the same total memory in bytes. Replay memory is counted at stored image size, and basis memory is counted at its encoded size including the full-precision outliers. All methods get the same hyperparameter search budget. Beating CLIB means a higher $A_{AUC}$ for CLIB-plus-our-memory than for CLIB alone at the same total bytes, across at least three budgets and five seeds. A result that matches CLIB at fewer bytes also counts.

Three further comparisons isolate the key ingredient. The first is QGPM run with fixed-interval pseudo-boundaries; this is the most natural port of the closest prior work. The second is our method with re-quantize-on-merge in place of append-only. The third is our method with uniform quantization in place of CINF. A full-precision version of our streaming memory gives the accuracy ceiling.

Why not just…?

Why not just run QGPM with a pseudo-boundary every N steps? This is the real threat. In i-Blurry, task segments have regular lengths, so a well-chosen N could approximate the hidden boundaries. We will sweep N over an order of magnitude and report the full curve, not the best point. We will also test on stream variants with irregular segment lengths, which we construct by resampling i-Blurry's task durations. If the best fixed interval matches our method on regular streams, we will report that honestly. The claim then narrows to robustness: our method should hold its performance when N is mis-set or segments are irregular, while the fixed-interval version should drop.

Why not just spend the bytes on more replay samples? At 4 bits, the 45 MB QGPM reports for ViT-B/16 still corresponds to a large number of stored images, and CLIB already shows that well-managed replay is strong. That is exactly why the equal-byte sweep is the core experiment rather than an afterthought. If the accuracy curve for replay alone dominates the curve for the mixed allocation at every budget, the proposal fails for this regime. We would report that result as evidence that projection memory only pays off when task boundaries exist.

Why not just quantize the basis uniformly, or keep it in full precision? QGPM reports that uniform quantization damages bases, because their heavy-tailed outlier entries carry the directional information. Full precision costs 515 MB on ViT-B/16 by its measurements. We include both as ablations rather than assume these results carry over to streaming. In particular, the full-precision streaming run tells us how much of any gap comes from quantization and how much comes from removing boundaries.

Proposed Experiments

The benchmarks are i-Blurry on CIFAR-10, CIFAR-100, and TinyImageNet, scored by $A_{AUC}$ and final accuracy. We add BlurryQ CIFAR-10, scored by final average accuracy and forgetting. For a Vision Transformer setting where basis memory is large, we use a blurry reordering of ImageNet-R built with i-Blurry's disjoint/blurry class split. No reported result was found on Lacuna for any blurry ImageNet-R stream. As a sanity check, we run 10-split CIFAR-100 with boundaries revealed, where the method should come close to QGPM and to GPM's reported 72.48% accuracy from the ROGO paper.

Two checks decide the outcome. First, the principal angle between the stored and shadow subspaces should stay flat over stream length for the append-only memory and grow for re-quantize-on-merge. If both stay flat, the core mechanism is unnecessary. Second, CLIB-plus-memory should beat CLIB alone at equal bytes on CIFAR-100 and TinyImageNet. If it does not, the method is a memory-saving trick for boundary-based learning and not an advance for blurry streams.

We will also report three secondary results. The persistence window should lower forgetting on blurry minor classes without hurting future classes. Accuracy versus N for the fixed-interval baseline should show the sensitivity described above. Finally, we will report wall-clock overhead per step.