Self-Checking One-Step Generators for Continual Replay

A standard fix for forgetting is generative replay. A model that learns tasks one after another tends to lose earlier tasks, so it trains a generator to re-create examples of old tasks and mixes those synthetic examples in with each new task. Diffusion models make good replay generators because they rarely collapse onto a few modes. They are slow, though, because each sample takes hundreds of sequential denoising steps. DDGR reports about 75,000 seconds of training on CIFAR-100, compared with about 8,000 seconds for regularization methods. One-step generators now come close to diffusion quality. FACM reports an ImageNet 256×256 FID of 1.70 from a single network call. Swapping the slow generator for a fast one is the obvious next move. That swap alone gives no warning when the fast generator starts to forget. A one-step generator that drifts produces confident, blurry, wrong replay data, and the "copy-of-a-copy" degradation that generative replay is known for returns.

We propose to use a structural feature of flow-anchored one-step generators. A single network outputs both a shortcut (a jump from noise straight to a sample) and an instantaneous velocity field (the local direction of the slow, many-step path from noise to data). In a model that has not forgotten, the two agree: integrating the velocity field for a few steps lands where the shortcut jumps. When they disagree on an old class, the shortcut has probably forgotten that class. The disagreement can be computed from noise alone, with no stored data. We use it for two things: as a data-free forgetting detector, and as a way to decide where to spend the scarce multi-step compute during replay.

Core Claim

Our hypothesis concerns which part of the network forgets first. FACM found that, even within a single dataset, training only the shortcut makes the network forget the instantaneous velocity field, which leads to exploding gradients. Supervising the velocity field as an "anchor" on a mirrored time interval restores stability. That paper motivates our design: it shows the shortcut is the fragile component and the velocity field is the stabilizer. We predict the same ordering under external task shifts. When a new class arrives, old-class shortcuts should degrade before the old-class velocity field does, because flow matching gives the velocity field a simple, non-bootstrapped regression target.

If the prediction holds, the gap between the two heads is informative. For a class label $c$ and noise $z$, let $\hat x(z,c)$ be the one-step output. Let $\tilde x_K(z,c)$ be the result of $K$ Euler steps along the model's own velocity head. We define the gap

$$g(z,c) = \lVert \hat x(z,c) - \tilde x_K(z,c) \rVert^2 ,$$

and its class average $G(c)=\mathbb{E}_z\, g(z,c)$.

The method works as follows. Bulk replay comes from one-step samples. Replay budget and $K$-step refinement are concentrated on classes with high $G(c)$ and on individual samples with high $g$. Old-class shortcuts are re-anchored to the velocity head with a consistency loss evaluated on noise under old labels, which again requires no stored data.

The new element is the use of within-model agreement between the two output heads as the control signal for replay. Neither DDGR, MAGD, nor the rank-1 Fisher work has an equivalent. The claim can fail in a clear way. If both heads drift together, $G(c)$ stays small while quality on old classes falls, and the detector has no value. That outcome is itself informative, and the first experiment is built to catch it.

Research Program

The base model is a class-conditional flow-anchored consistency model in the FACM style, trained with a flow-matching anchor on a mirrored time interval. We also run a Mean Flow variant (Mean Flows trains the average velocity directly and gets FID 2.92 on CIFAR-10 in one step). For this variant, the velocity head comes from evaluating the average velocity at $r=t$. Each new task is initialized with CMT-style mid-training on short trajectories, so the shortcut is not bootstrapped from scratch. CMT was validated only on static datasets, and we reuse its method as-is.

The research proceeds in three stages.

The first stage is purely diagnostic. During sequential class-incremental training with naive one-step replay, we track per-class FID and classifier accuracy on true held-out old-class data, together with $G(c)$, separately for each head. The question is which head forgets first, and whether $G(c)$ ranks classes by true degradation.

The second stage builds the gap-guided replay allocator and the data-free re-anchoring loss. We sweep the number of refinement steps $K$ from 2 to 16 and the fraction of samples that get refined.

The third stage tests whether the low-noise-level geometry found by Rank-1 Fisher carries over to the velocity head of a flow-map model. That paper showed that at high noise the denoiser acts almost linearly, per-sample gradients line up in nearly the same direction, and the curvature estimate needed for EWC (elastic weight consolidation) collapses to a single direction. EWC is the method of penalizing changes to parameters that mattered for old tasks, weighted by a curvature estimate. The paper left flow objectives and the average-velocity derivative term untested. If the rank-1 structure survives on the velocity head, we add rank-1 EWC to that head only. This protects the anchor at little cost and leaves the shortcut free to adapt.

Baselines

The naive reference is fine-tuning the one-step generator on each new task with no replay and no regularization. It sets the forgetting floor.

The first strong comparison is DDGR, which uses classifier-guided multi-step diffusion replay. We run it at its original step count and with DDIM at 10 steps. DDGR's reported numbers are 59.20% accuracy on CIFAR-100 (AlexNet, 5 classes per increment) and a 6.98% forgetting rate on ImageNet with 50 classes per increment, against 17.31% for GAN-based generative replay.

The strongest existing generator-side method is rank-1 EWC plus teacher-student generative distillation from Rank-1 Fisher. That paper reports that it "more than halved the forgetting compared to using generative distillation alone" on 20-task ImageNet-1k. No numeric FID was found on Lacuna for that result, so we will reproduce it using the released code (rank-1 Fisher codebase). We also include MAGD, which distills from intermediate noisy states with an EMA (exponential moving average) teacher. MAGD reports a 92% compute saving over 1000-step DDGR, but no absolute accuracy or FID numbers were found on Lacuna. For classifier retention without a generator, PASS++ reports 53.43% last accuracy on CIFAR-100 over 20 phases.

The closest threat is our own ablation: the same one-step generator with EMA-teacher replay and a uniform budget, with no use of the gap. A second ablation spends the same refinement budget on randomly chosen samples. A third selects samples by classifier confidence instead of by $G$.

All methods share the same backbone, task splits, and total generator-plus-classifier GPU-hour budget. Each method receives the same number of hyperparameter trials. Beating the field means matching or improving on DDGR's accuracy and forgetting at no more than one tenth of its wall-clock time. Isolating the contribution of the gap means beating the uniform-budget and confidence-filter ablations on old-class FID at equal network-call counts.

Why not just…?

Why not just plug a 5–10 step DDIM sampler into DDGR? DDGR applies classifier guidance at every reverse step, and coarse steps amplify discretization error in exactly the samples that feed the next generator. FACM reports that Euler sampling on curved trajectories needs around 250 steps, about 7 seconds per image against 70 ms for its 2-step sampler. We still run DDIM-10 DDGR as a baseline. If it matches our accuracy at similar cost, the one-step machinery is unnecessary, and we will report that.

Why not just use a one-step generator with EMA-teacher replay and skip the gap signal? This is the real threat, and it is the main ablation. The gap has to earn its place by predicting which classes degrade. The Stage 1 correlation test and the uniform-versus-gap comparison at a fixed call budget settle the question directly. If uniform replay performs just as well, we will report that the detector adds nothing beyond monitoring.

Why not just filter replay samples by classifier confidence? A confident but blurry sample passes a classifier filter and still poisons the next generator. That is the copy-of-a-copy mechanism DDGR describes. The gap measures generator disagreement rather than label certainty. We test this head-to-head as the confidence-filter ablation.

Why not just store exemplars or use diagonal EWC? When exemplars are allowed, they remain the right answer, and we report an exemplar upper bound. Our target setting is the data-free one motivated by MAGD and FedCIL. Rank-1 Fisher measures the diagonal Fisher error in diffusion networks at "near 100%," which makes diagonal EWC a weak competitor. We include it anyway for completeness.

Proposed Experiments

We use three benchmarks. The first is CIFAR-10 class-conditional generation split into 5 tasks of 2 classes each, following the Rank-1 Fisher protocol. The second is CIFAR-100 class-incremental classification with generator replay, at 5 and 20 phases, following DDGR and PASS++. The third is 20-task downsampled ImageNet-1k. We report per-task old-class FID, average accuracy, forgetting rate, number of network calls, and wall-clock time.

Four checks decide the outcome. First, on CIFAR-10, the Spearman correlation between $G(c)$ and true old-class FID increase must be clearly positive across tasks. If it is not, both heads drift together and the method is abandoned in favor of a pure latency study. Second, velocity-head FID from $K$-step integration on old classes must degrade more slowly than shortcut FID, which confirms the forgetting ordering the method relies on. Third, at equal network-call budgets, gap-guided replay must beat both uniform and confidence-filtered replay on old-class FID. Fourth, on CIFAR-100 and ImageNet, the full system must reach DDGR-level accuracy and forgetting at no more than one tenth of DDGR's wall-clock time.

As a side result, we will measure whether per-sample gradients of the velocity head at high noise remain nearly collinear. This tells us whether rank-1 EWC transfers to flow-map models.