Paying for Synthetic Replay Only When the Backbone Moves

Continual learners that cannot store real data, whether for privacy reasons or because the original data is out of reach, often replay synthetic images instead. They generate these images with a diffusion model, a generator that learns by gradually turning random noise into samples. That generation is expensive, and current methods spend it uniformly. Every task gets the same number of images, and those images are used to regularize the whole network. Long-tailed and incremental learning offer a different picture: much of forgetting is a classifier problem, not a feature problem. If the feature extractor (the backbone) is frozen, the classifier can be re-balanced from cheap stored feature statistics, and no images are needed at all.

We propose treating the stability-plasticity decision as a decision about compute. In each new session, the learner measures how badly its frozen features serve the new classes. If they serve them well, it keeps the backbone frozen and recalibrates the classifier from stored statistics at almost no cost. If they serve them poorly, it unfreezes the backbone and buys only as many synthetic "anchor" images as it needs. These anchors do two jobs: they limit forgetting while the backbone trains, and they re-estimate old-class feature statistics after the backbone has moved. The core bet is that the right number of anchors depends on how unevenly the backbone moves, not on how many classes or tasks have been seen. That quantity is cheap to measure.

Core Claim

In a two-stage learner, the backbone is trained or frozen first, and the classifier is then recalibrated on class-balanced inputs. In that setting, synthetic images do no useful work while the backbone is frozen. Their value comes from two places: restraining how the backbone drifts, and refreshing the stored old-class feature means and covariances after it drifts.

The second job has a simple sample-size requirement. Suppose an update moves each old-class feature by a displacement $\delta$. The class mean can be corrected from $n_c$ anchors with standard error roughly $\sqrt{\mathrm{tr}(\Sigma^{\Delta}_c)/n_c}$, which follows from the usual standard error of a sample mean. Here $\Sigma^{\Delta}_c$ is the covariance of the per-sample displacements within class $c$. So

$$ n_c \approx \frac{\mathrm{tr}(\Sigma^{\Delta}_c)}{\varepsilon^2}, $$

where $\varepsilon$ is the tolerated error in the corrected mean.

This makes a falsifiable prediction. A large but rigid drift needs few anchors. A small but incoherent drift needs many. A uniform budget will overspend in the first case and underspend in the second. Our claim is that a controller built on this rule matches fixed-budget synthetic replay at a fraction of the generation cost, and does better than both "always freeze" and "always fine-tune" on hard class orders.

How This Relates to Prior Work

Two continual-learning methods with synthetic replay motivate the cost problem, and both spend their budget uniformly and end to end. Synthetic Data is an Elegant GIFT for Continual Vision-Language Models generates 1,000 Stable Diffusion images per task. It uses them to distill a CLIP model end to end, with Fisher-based adaptive weight consolidation, and notes that diffusion overhead remains severe. CD^2 distills a fixed number of synthetic "caricatures" per class for few-shot incremental learning. It raises its retention-loss weight as classes accumulate, updates the backbone and classifier jointly, and reports that synthesis cost stays high. We keep their synthetic sources but change what the images are used for and how many are made.

Three papers supply the decoupling mechanism. Decoupling Representation and Classifier for Long-Tailed Recognition shows that a frozen backbone plus simple classifier recalibration beats complex joint training. DELTA carries the same split into online continual learning, but relies on a buffer of real exemplars. Lightweight Federated Incremental Learning via Decoupled Replay is the closest existing work. It replays generated 512-dimensional features into the classifier and regularizes the encoder separately, but with static loss weights and no rule for when the encoder should move or what that movement costs. We differ from all three in making the freeze/unfreeze choice and the generation budget per-session quantities tied to measured drift.

Two dynamic-balancing papers supply the feedback idea, though neither lets the signal control generation spending. MEGA rescales gradients between current and memory losses. Asymmetric Synthetic Data Update learns per-image stability/plasticity weights through expensive bi-level optimization over pixels, and handles only a fixed label space.

Two results constrain the design. Rectified Decoupled Dataset Distillation finds that random real images with teacher soft labels beat almost all costly distillation on ImageNet-1K. We therefore make the anchor source interchangeable and test real anchors directly. Prioritized Generative Replay reports degradation once synthetic data passes 87% of the buffer. This argues for using synthetic images as sparse anchors rather than as bulk training data. Finally, The Lie of the Average shows that averages over random class orders hide collapses. We therefore report worst-case orders.

Research Program

The learner keeps, for each old class, a feature mean and a low-rank covariance, as a few hundred floats per class. At the start of each session it runs a cheap plasticity probe. It fits a linear head on frozen features of the new classes for one epoch and compares that head's training loss and nearest-class-mean accuracy against a threshold calibrated on the first session only. This removes the need for a validation stream.

If the probe passes, the backbone stays frozen. The classifier is retrained on real new-class features plus Gaussian samples drawn from the stored old-class statistics. This costs no generation.

If the probe fails, the learner makes a short trial update of the unfrozen upper blocks. It generates a small pilot set of anchors per old class, for example 8 images, and measures $\mathrm{tr}(\Sigma^{\Delta}_c)$ from their feature displacement. It then tops up anchors per class according to the rule above. It uses the full anchor set twice: first as a feature-distillation constraint during the backbone update, then to re-estimate class statistics before the classifier is recalibrated. Classes whose displacement is coherent receive almost no extra images.

The analysis component tests the sample-size rule directly. For each session we will plot the realized error in the refreshed class means against predicted anchor counts, and we will report where the rule breaks, for example when displacement is heavy-tailed. We will also track how generation cost accumulates across sessions under each policy.

Baselines

The naive reference is freeze-and-recalibrate. The backbone is trained on the first session and never updated. The classifier is recalibrated each session using nearest-class-mean and Gaussian samples from stored statistics, which is the classifier-recalibration regime of Kang et al.. It costs nothing to generate and is our lower cost bound. Our method must match or beat it on standard orders, and clearly beat it on hard orders where new classes are poorly served by old features.

The strongest synthetic competitor is uniform end-to-end synthetic replay. We implement it in two ways. In the few-shot incremental protocol, we reimplement CD^2 as reported. It reports 68.67% average accuracy and 57.36% final-session accuracy on CIFAR-100, against iCaRL at 13.73%, and we quote these figures from CD^2 and reproduce them before comparing. In standard class-incremental learning, we run a GIFT-style variant with a fixed number of Stable Diffusion images per task distilled into the whole network. No reported result was found on Lacuna for GIFT's exact CIFAR-100 class-incremental breakdown, so we use our own reproduction. Both baselines receive the same backbone, the same generator, the same training epochs, and an equal hyperparameter search. Beating them means equal or better final and worst-order accuracy at no more than half the total GPU-hours counted end to end, including generation.

The strongest real-data competitor is random real exemplars with soft labels, following Rectified Decoupled Dataset Distillation, with a buffer matched in image count to our average anchor spend. We also compare against DELTA-style decoupled training with real exemplars in its own long-tailed setting. There DELTA reports 19.93% average accuracy on CIFAR-100-LT with 20 tasks, a 2K buffer, and imbalance factor 0.01, as stated in DELTA.

Two ablations isolate the key ingredient. The first gives the decoupled pipeline the same total anchor budget as our method, spread uniformly across sessions and classes. If it matches us, the drift-based allocation adds nothing. The second always unfreezes the backbone but keeps the variance-based allocation, which isolates the value of the freeze decision.

Why not just…?

Why not just store real exemplars with soft labels? RD^3's evidence makes this the most serious threat. Where storage is allowed, it may simply win. Our method treats the anchor source as interchangeable, so we will run the controller with real anchors as well. That tests whether allocation helps regardless of source, and whether synthetic anchors approach real ones at matched count. The synthetic arm is justified only in the privacy-restricted regime that Li-FIL describes. We will report plainly if real anchors dominate everywhere storage is permitted.

Why not just replay Gaussian features from stored statistics and never generate images? This works exactly as long as the backbone does not move, and it is our freeze-and-recalibrate baseline. Once the backbone updates, the stored statistics describe a feature space that no longer exists. The hard-order experiments show how much accuracy is lost as a result, and the mean-refresh error plots show how quickly the stored statistics go stale.

Why not just train end to end and apply cosine or τ-normalization afterward? Hou et al. and Kang et al. show that such fixes remove magnitude bias. However, τ needs a validation set that streams rarely provide, and DELTA reports that joint online training distorts features before any post-hoc fix can act. We include end-to-end training with cosine normalization, with τ tuned on held-out data as an oracle advantage. If it matches us even with that advantage, the decoupling adds nothing.

Proposed Experiments

Our main benchmarks are CIFAR-100 class-incremental learning in 10 and 20 sessions, CIFAR-100 few-shot incremental learning under the CD^2 protocol, CUB-200, and ImageNet-R. For CIFAR-100 we evaluate each method on three random class orders and on the adversarial semantic orders from The Lie of the Average. That paper reports EWC falling from about 26% to about 12% on these orders. On ImageNet-R, the memory-free comparison point is PromptFusion's reported 78.7% from PromptFusion. We use it only as context, because that method never fine-tunes its backbone.

We report final and average accuracy, worst-order accuracy, forgetting, total GPU-hours including generation, and the number of generated images. We present these as accuracy-versus-compute curves rather than single points.

Two checks are decisive. The first is the uniform-budget ablation at equal image count. If drift-based allocation does not beat it by a margin larger than the variation across class orders, the core claim fails. The second is the anchor-count prediction. Across sessions, the per-class variance of displacement should predict the anchors needed to refresh the class means to a fixed tolerance better than displacement magnitude or class count does. If magnitude predicts as well, the rule reduces to simpler drift-triggered schedules such as CD^2's growing weight, and we would recommend those instead.