When a model learns tasks one after another, it tends to forget the earlier ones. A standard fix is generative replay: a generator trained on old data produces imitation old examples, which are mixed into training on each new task. A generator, however, can memorize. If an evaluation item leaked into an early task's data, a replay generator can keep reproducing it, task after task. The learner then keeps re-studying the exam, and the measured "retention" of old tasks partly reflects a memorized answer key. We propose to measure this replay-carried contamination directly. We also propose a remedy with a provable ceiling: train the replay generator with differential privacy (DP), a guarantee that bounds how much any single training record can change the generator's output distribution. We train it in a compressed latent space, which is what makes private generators good enough to be useful.
The two-sentence version is this. Generative replay can carry leaked test items forward through a continual-learning run and inflate retention scores. A replay generator trained privately in a low-dimensional latent space provably limits how much any one leaked item can reappear, and so caps that inflation while keeping most of replay's benefit against forgetting.
Core Claim
Continual-learning evaluation assumes the replay stream carries knowledge about old tasks, not specific old records. Nothing enforces that assumption. Two strands of evidence motivate the worry. Hazard Compression shows that diffusion-based replay in safe reinforcement learning distorts its own distribution through feedback: the share of rare hazardous transitions the generator still produces falls from 13.3% to 0.6% on Cheetah-run. Replay generators are therefore not faithful copies of the past. Separately, the contamination literature shows that static scores often reflect memorized items. DyVal finds models with strong static scores near zero on freshly generated reasoning problems, and Snoopy ties few-shot performance to how often terms appeared in pretraining. No work in the file asks whether replay transports contamination across tasks.
Our claim is twofold. First, with a non-private generator, accuracy on leaked evaluation items survives later tasks much better than accuracy on matched clean items does, and the gap grows with how many times the item was duplicated. Second, a replay generator trained with DP-SGD bounds that gap through the guarantee itself. DP-SGD trains with per-example gradient clipping and Gaussian noise. An item present $k$ times enjoys group privacy at level roughly $k\varepsilon$. Anything computed purely from replay samples, including a learner's accuracy on that item, can therefore shift by at most a factor of about $e^{k\varepsilon}$ plus a $\delta$-scaled term. The second claim is only practical because of dimensionality reduction. Pixel-space private generators are poor. Differentially Private Latent Diffusion Models (DP-LDM) privately fine-tune only the attention blocks of a latent diffusion model with a frozen public autoencoder, and reach FID 8.4 on CIFAR-10 at $\varepsilon=10$ in 15 GPU hours. DP-LFlow similarly trains normalizing flows privately on autoencoder latents. Both papers supply our method. Neither ever used it in a sequential setting, and neither framed DP as a contamination control.
One constraint shapes the design from the start. DECO-Bench shows that when only a small adapter is modified, the frozen base weights still expose information the adapter was meant to remove. DP on the fine-tuned attention blocks says nothing about what the frozen public autoencoder and backbone already memorized. We therefore plant contamination only in the private task stream, and we separately audit the public backbone. We do not claim protection we cannot prove.
Research Program
The core testbed is class-incremental image classification on split CIFAR-10 and 20-Split miniImageNet, the latter as used by Shared and Private VAEs. Contamination is controlled by planting. For each task we draw a set of test images and insert them into that task's training data at duplication counts $k \in \{1, 4, 16\}$. Each planted item is paired with a clean "twin" test image of the same class that never enters training, matched on the learner's initial confidence. The quantity we track is contamination inflation after task $t$:
$$\Delta_t = \mathrm{Acc}_t(\text{planted}) - \mathrm{Acc}_t(\text{twins}),$$
where $\mathrm{Acc}_t$ is the learner's accuracy on the indicated items after training through task $t$.
Right after a task is learned, $\Delta$ is high for every method, because the learner saw the planted items directly. The question is what happens over later tasks. Without replay, $\Delta$ should decay as the learner forgets. With a non-private replay generator, we predict that $\Delta$ persists or grows, because memorized copies keep being replayed. With DP latent replay, we predict that $\Delta$ decays to near the no-replay curve while overall old-class accuracy stays close to non-private replay. The ordering of $\Delta$ across $k$ and $\varepsilon$ tests the group-privacy prediction.
There is a subtlety: the learner's own weights carry memorization independently of the replay channel. To test the bound cleanly, we add a "replay-only" probe. After task $t$, a fresh classifier is trained solely on replay samples. The DP guarantee applies exactly to this classifier as post-processing, so its $\Delta$ must stay inside the theoretical envelope. Violating the envelope would indicate an accounting bug, which makes the probe a check on the implementation as well. We also run a nearest-neighbor regurgitation audit on replay samples in the autoencoder latent space and in a separate public feature space. This measures how often generated images fall within a small radius of planted items.
The generator follows DP-LDM. At each new task we privately fine-tune only the attention blocks of a public latent diffusion model on the current task's data, conditioned on class. Old tasks are represented by the generator snapshots each produced. Budgets therefore compose in parallel across disjoint task data, the same parallel-composition argument DP-LFlow uses for class-partitioned generators. The generator never retrains on its own outputs, which avoids the self-consuming loop that Hazard Compression documents. We sweep $\varepsilon \in \{1, 4, 10\}$. The file notes that fidelity drops sharply at $\varepsilon \le 1$, so the low end tests where the trade-off breaks.
A second, smaller study tests the phenomenon outside our planted setting. Capacity-Gated Forgetting reports that endogenous replay, where the base LLM samples its own text as rehearsal, cuts average MMLU forgetting from 0.352 to 0.070 at one seed during MedQA fine-tuning. The same paper reports the gain collapses to +0.005 at two other seeds, and its authors cannot tell format calibration from knowledge loss. MMLU is widely present in web data. If the base model's self-samples contain MMLU-like question strings, part of the measured mitigation could be re-rehearsal of the test. We will repeat their QLoRA setup on Qwen3.5-9B with the batch size and sequence length fixed across at least five seeds. We will measure n-gram and embedding-level overlap between rehearsal samples and MMLU items. We will compare forgetting on original MMLU items against reformatted twins with permuted options and paraphrased stems, and against DyVal problems. This study involves no privacy mechanism. Its job is to show whether replay-carried contamination exists in a published result, which motivates the controlled study.
Baselines
The naive reference is fine-tuning with no replay. It defines both the forgetting floor and the natural decay curve of $\Delta$. Any replay method must beat it on old-class accuracy. A replay method that "beats" it on $\Delta$ by keeping $\Delta$ higher is exhibiting the failure we are measuring, not succeeding.
The strongest existing replay method on these benchmarks is the shared-plus-task-specific VAE replay of Shared and Private VAEs. It reports 77.4% accuracy on 20-Split miniImageNet under restricted synthetic generation and 100% with ample generation, using a 0.8 MB footprint. Its "private" means task-isolated, not differentially private. We will rerun its released configuration under our planted-contamination protocol with matched generated-sample counts and epochs. DP latent replay beats it if it keeps old-class accuracy within a few points of this baseline under the same generation budget while holding $\Delta$ at the no-replay level after the final task. For DP-generated replay on split CIFAR-10 or miniImageNet, no reported result was found on Lacuna. The closest anchor is DP-LDM's single-task FID 8.4 on CIFAR-10 at $\varepsilon=10$.
A raw exemplar buffer is a second strong competitor. We give it the same per-task budget measured in bytes, and also run a version where the learner is trained on the buffer with DP-SGD. This tests whether privatizing the learner is enough. Elastic Weight Consolidation stands in for regularization methods that replay no samples. It cannot regurgitate items, so it gives a no-replay-channel reference for $\Delta$ that keeps more accuracy than plain fine-tuning.
Two ablations isolate the proposal's ingredients. A non-private latent diffusion generator with an identical architecture and fine-tuned blocks isolates the effect of DP noise. A pixel-space DP diffusion generator with the same $\varepsilon$ and comparable compute isolates the role of the latent bottleneck. We also test DP LoRA in place of attention fine-tuning, given DP-LDM's report that LoRA underperforms under DP noise. Every method gets the same data, the same total GPU budget per task, and the same number of hyperparameter trials, with hyperparameters tuned on clean held-out data only. All runs use five seeds, given the seed fragility documented in Capacity-Gated Forgetting.
Why not just…?
Why not just filter replay samples against the test set with nearest-neighbor or n-gram matching? This is the real threat in our planted setting, because there we know exactly which items to filter. We include a non-private replay arm with latent and pixel nearest-neighbor filtering at several radii. The filter needs the evaluation set to be known at training time, which fails for private or future benchmarks. It also misses near-duplicates and augmented variants. We test this directly by planting flipped, cropped, and color-jittered copies. If filtering matches DP replay on $\Delta$ in every condition, we will report that the practical contribution reduces to the guarantee that holds without knowing the test set. The file's evidence that exact-match decontamination misses semantic shortcuts (Back to Square One, Snoopy) suggests filtering will leak on variants, but the experiment decides.
Why not just evaluate continual learners on dynamic benchmarks like DyVal and leave replay alone? Dynamic evaluation detects inflation, and we use it that way in the LLM study. It does not stop a replay stream from re-teaching leaked items, so a deployed learner still overfits them. It also requires formally specified tasks, as the file notes for DyVal and P⁴Bench. Image class-incremental learning has no such generator. Measurement and prevention complement each other here.
Why not just train the learner with DP-SGD instead of the generator? A privatized learner bounds memorization in its weights, but it pays the high-dimensional noise cost on every update of every task. Its retention also still depends on whatever the replay source supplies. We run this arm. We expect it to lose more old-class accuracy than DP latent replay at matched $\varepsilon$, for the same dimensionality reason that DP-LDM and DP-LFlow document for generators. If it does not, the latent bottleneck is not what matters, and we will say so.
Decisive Checks
Three results would confirm the proposal. Non-private replay should show $\Delta$ after the final task that stays well above the no-replay curve and rises with duplication count $k$. DP latent replay should bring final $\Delta$ to within seed noise of no-replay and EWC at $\varepsilon \le 4$, while keeping old-class accuracy within a few points of non-private replay. The replay-only probe should keep $\Delta$ inside the $e^{k\varepsilon}$ envelope at every $k$.
Three results would refute or narrow it. If non-private replay shows no persistent $\Delta$, replay-carried contamination is not a real problem. If DP replay suppresses $\Delta$ only at budgets where old-class accuracy collapses, the cure costs too much. If the frozen public backbone already reproduces planted-like items, the guarantee covers too little of the risk. The LLM study then indicates whether these effects plausibly explain part of the seed-42 endogenous-replay gain reported in Capacity-Gated Forgetting. That would show up as a larger mitigation gain on original MMLU items than on reformatted twins and DyVal problems.
References
- Hazard Compression: Catastrophic Forgetting in Diffusion-Based Generative Replay under Distribution Shift
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Snoopy: An Online Interface for Exploring the Effect of Pretraining Term Frequencies on Few-Shot LM Performance
- Differentially Private Latent Diffusion Models
- DP-LFlow: Differentially Private Latent Flow for Scalable Sensitive Image Generation
- DECO-Bench: Unified Benchmark for Decoupled Task-Agnostic Synthetic Data Release
- Shared and Private VAEs with Generative Replay for Continual Learning
- Capacity-Gated Forgetting in LoRA Fine-Tuning: Rank, Proximity, and Endogenous Replay in Medical LLMs
- Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schema
- $\text{P}^4$Bench: Contamination-Proof, Publicly Verifiable, and Privacy-Preserving LLM Evaluation via Zero-Knowledge Proofs