When a neural network learns a sequence of tasks, it tends to forget earlier ones. A common fix that avoids storing old data is generative replay: a generator learns to produce imitation samples of past data, and those samples are mixed into later training. The imitation samples are imperfect, so this fix brings in its own error. This note argues that two separate causes of forgetting are usually lumped together. The first is interference, where new learning overwrites old weights, and it happens even when replay is perfect. The second is noise, the extra damage done by replaying blurry or hallucinated samples. A wireless engineer would recognise the split. Link quality depends on signal over interference plus noise, and the common shortcut of ignoring noise, called the "interference-limited" assumption, fails exactly in sparse regimes. Continual learning methods make the same shortcut without saying so. They treat forgetting as a gradient-conflict problem only.
The split matters most when skills depend on one another. Students who know only one of two skills needed for a problem perform almost as if they knew neither. A Bayesian model fitted to tutoring data converges to an AND-gate: effective knowledge is about 0.05 with one of two skills and about 1.0 with both (The Composition Effect). If composite skills in neural learners behave the same way, a small fidelity loss on one foundational skill can wipe out every composite built on it. That loss would be nearly invisible to any flat, per-class quality measure. We propose to measure, for each skill, how much of its forgetting is interference and how much is replay noise. We then spend a small budget of real stored examples, which we call anchors, where noise is high and many downstream skills depend on the result.
Core Claim
Under a fixed memory budget, allocating a tiny real-exemplar buffer by the product of two quantities beats uniform allocation, forgetting-rate allocation, and pure real or pure generative replay on streams of hierarchically dependent skills. The first quantity is a skill's estimated noise share of forgetting. The second is its conjunctive importance to downstream composites.
A second, falsifiable claim concerns synthetic replay volume. More imitation samples should help skills that are interference-limited and hurt skills that are noise-limited. This would explain the "volume paradox" reported in CAN, where enlarging a low-quality synthetic buffer first lowers accuracy. It would also turn that paradox into a per-skill prediction.
The Decomposition
Fix a skill $k$ and the end of a task stream. Let $e_k^{\text{real}}$ be its error when old data is replayed from true samples, $e_k^{\text{gen}}$ its error under generative replay at equal sample count, and $e_k^{0}$ its error immediately after it was learned. We define interference $I_k = e_k^{\text{real}} - e_k^{0}$ and noise $N_k = e_k^{\text{gen}} - e_k^{\text{real}}$. A skill is interference-limited when $I_k \gg N_k$ and noise-limited otherwise.
The full decomposition needs oracle runs, so it serves only as an analysis tool. The deployable method uses a cheap online proxy. For each skill it keeps a handful of real examples, and the gap between the solver's loss on those examples and its loss on generated examples of the same skill estimates $N_k$.
For a composite skill $c$ with prerequisites $\text{pre}(c)$, the AND-gate hypothesis predicts that accuracy factorises:
$$a_c \approx \prod_{k \in \text{pre}(c)} a_k ,$$
where $a_k$ is the accuracy on prerequisite $k$. Under this form, the gain in composite accuracy from fixing prerequisite $k$ scales with the product of the other prerequisites' accuracies. This gives the allocation score
$$s_k = \hat N_k \sum_{c:\,k \in \text{pre}(c)} \prod_{j \in \text{pre}(c),\, j \neq k} a_j ,$$
where $\hat N_k$ is the online noise estimate. Real anchor slots go to the highest-scoring skills, and generated samples fill the rest of each replay batch. The factorisation is itself a hypothesis to test. If composites turn out to be compensatory in networks, the product collapses toward a sum and the hierarchy weighting should stop mattering. Our ablations would show that.
Relation to Prior Work
Several papers motivate the gap. Continual Learning with Deep Generative Replay introduced the generator-plus-solver "scholar" and kept MNIST accuracy at 98.56% after five generations. Three types of incremental learning found that the approach struggles on CIFAR-100 and that small real buffers often win. Fidelity failures in sequence domains, where self-generated text lags real exemplars (Mi et al., Qin et al.), suggest the noise term is large exactly where compositional skills live. All of these treat fidelity loss as uniform across classes. We instead ask where it lands in a dependency structure.
Two papers come closest in method. Autoencoder-Based Hybrid Replay compresses exemplars into latents and separates class centroids. CAN allocates replay by client-specific forgetting rates. CAN is the closest prior allocation scheme, but it weights by how much is forgotten, not why it is forgotten or what depends on it. Our ablations separate those factors.
Other work treats interference without noise. Meta-Experience Replay pushes current and past gradients toward positive inner products. GMED edits stored examples to maximise interference. Both are interference-only by construction: they assume replay samples are faithful. Our decomposition tells us when that assumption holds.
Hierarchy work supplies the dependency structure. Skill-it! shows that ordering data along a skills graph improves sample efficiency. Skill Neologisms composes isolated soft-token skills zero-shot. Neither studies forgetting of shared parameters under imitation replay. The wireless papers are a lens, not a method. Vaze shows that denser transmitters can destroy connectivity, and Bao et al. show the noise-free model breaking in sparse regimes. Together they motivate testing the noise-free assumption rather than adopting it. We borrow no wireless mathematics beyond the signal-over-interference-plus-noise decomposition.
Research Program
We would build a continual stream of compositional skills on the synthetic digit-sequence transformation tasks used by Skill Neologisms. Atomic skills such as sorting, reversing, and adding one arrive first. Two- and three-skill composites arrive later, and atomic skills are never revisited with real data except through the anchor buffer. A small transformer serves as the solver. Pseudo-samples come in two forms. The first is self-generated inputs labelled by the previous solver, following the scholar recipe. The second is a separate small generator. Using both checks that the findings are not specific to one generator.
We would repeat the design on LEGO reasoning chains, where dependency depth is controllable. As a flat control, we would run split CIFAR-100 class-incremental learning. There the hierarchy term is constant, and the method should reduce to noise-weighted allocation.
The analysis phase computes the oracle decomposition $(I_k, N_k)$ per skill. It then tests whether the product form above predicts composite accuracy better than an additive (compensatory) form, scored by held-out fit. Finally, it checks whether the online proxy $\hat N_k$ correlates with the oracle $N_k$. The method phase applies the allocation score with anchor budgets of a few to a few dozen examples per skill.
Baselines
The naive reference point is fine-tuning with no replay, which bounds forgetting from above. At the other end, real replay with an unlimited buffer gives the oracle that defines $I_k$.
The strongest simple competitor is standard experience replay with reservoir sampling, given the entire memory budget in raw examples. Three types of incremental learning reports that this often beats sophisticated methods. On split CIFAR-10, GMED reports it at 33.30%, rising to 34.84% with GMED and 47.47% with augmentation. We would run experience replay and experience replay plus GMED with identical byte budgets, counting the generator's parameters against our method's budget. That is the condition under which pure real replay is hardest to beat.
The strongest existing hybrid is AHR, which reports 54.43% final accuracy on CIFAR-100 class-incremental learning against 49.38% for iCaRL. On the CIFAR-100 control we would reproduce it at equal memory and require our method to match it within run-to-run variance. Since that benchmark has no hierarchy, we would not claim a win there. On the compositional stream, no reported result was found on Lacuna for any generative-replay method. All numbers there would come from our own runs under one shared tuning budget, with equal hyperparameter search trials per method and equal solver training steps.
Plain generative replay with no anchors is the method's own starting point. Three ablations isolate the key ingredients. Uniform anchors spread the buffer evenly across skills. Forgetting-rate anchors, in the style of CAN, allocate by total forgetting. Noise-only anchors use $\hat N_k$ without the hierarchy weight. Beating the forgetting-rate ablation on composite accuracy, at equal anchor count, across at least three seeds with non-overlapping confidence intervals, is the result that would establish the contribution.
Why not just…?
Why not just store real examples and skip the generator? This is the most serious threat. At small budgets it may simply win. The equal-byte comparison above settles it. If experience replay matches our method across budgets on the compositional stream, the generator adds nothing and we would report that. Our expectation rests on AHR's observation that compressed storage fits roughly ten times more examples per byte. Where memory is tight, the generator's coverage should matter, and anchors would fix the generator's worst failures.
Why not just build a better generator? Uniformly better fidelity, for example from diffusion models, shrinks $N_k$ everywhere but costs heavy sampling compute. The note's claim is about where residual noise lands, not about eliminating it. We would test this by swapping in a stronger generator. If hierarchy-weighted anchors still help on composites, the allocation is complementary to fidelity. If the gain vanishes, fidelity alone suffices in that regime, and the decomposition would show $N_k \to 0$.
Why not just allocate by forgetting rate, as CAN does? Forgetting rate mixes interference with noise. Real anchors do not cure interference: real replay at equal count is exactly the condition under which $I_k$ remains. Forgetting-rate allocation should therefore waste slots on interference-limited skills. It is an explicit ablation, and if it ties our method, the regime decomposition is unnecessary.
Why not just align gradients with MER? Gradient alignment attacks $I_k$ and assumes replay samples are faithful. It cannot fix a skill whose replayed inputs are wrong, and it costs multiple gradient evaluations per step. We would run MER on top of generative replay. If noise-limited skills remain broken under MER while anchors repair them, the two failure sources are distinct.
Decisive Checks
Three results would decide the proposal. First, composite accuracy should be predicted better by the product of prerequisite accuracies than by their sum; if not, the hierarchy weighting is unmotivated. Second, sweeping synthetic replay volume should produce rising curves for skills with low $\hat N_k$ and the non-monotonic dip for skills with high $\hat N_k$, reproducing the volume paradox per skill rather than in aggregate. Third, at equal memory, the full allocation should beat both forgetting-rate anchors and experience replay on three-skill composites. If the first check fails, we would simplify the method to noise-weighted anchors. If the third fails against experience replay, we would conclude that generative replay is not worth its cost for compositional skills at these scales.
References
- The Composition Effect: Conjuntive or Compensatory? An Analysis of Multi-Skill Math Questions in ITS
- CAN: Leveraging Clients As Navigators for Generative Replay in Federated Continual Learning
- Continual Learning with Deep Generative Replay
- Three types of incremental learning
- Continual Learning for Natural Language Generation in Task-oriented Dialog Systems
- Lifelong Sequence Generation with Dynamic Module Expansion and Adaptation
- Autoencoder-Based Hybrid Replay for Class-Incremental Learning
- Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference
- Gradient-based Editing of Memory Examples for Online Task-free Continual Learning
- Skill-it! A data-driven skills framework for understanding and training language models
- Skill Neologisms: Towards Skill-based Continual Learning
- Percolation and connectivity on the signal to interference ratio graph
- Understanding the benefits of open access in Femtocell networks: stochastic geometric analysis in the uplink