When a deep neural network is trained on a sequence of tasks or shifting data distributions, it suffers from a fundamental flaw: it tends to completely overwrite what it previously learned. This phenomenon, known as catastrophic forgetting or catastrophic interference, occurs because standard optimization algorithms adjust weights solely to minimize the loss on the current data distribution.
To build an intuitive understanding of why models forget and how modern research prevents it, we can look at the problem through three distinct lenses: representation alignment, parameter isolation, and strategic memory rehearsal.
The Mechanics of Forgetting: Semantic Drift and Representation Mismatch
Why does a model forget? When a network is fine-tuned on a new domain or task, the optimization process is entirely blind to past objectives. As a result, the shared parameter space undergoes what Yoon et al. describe as semantic drift—global weights move away from the optimal configurations required for earlier tasks.
From a theoretical perspective, Zheng et al. frame this as an identifiability problem. When a model trains on a sequence of partial tasks, its latent representations become "partial-task aware" rather than "all-task aware." The network optimizes its internal features solely to distinguish between the classes or features present in the current batch, inadvertently collapsing or scrambling the latent boundaries established during previous training phases.
Constraining the Updates: Knowledge Distillation and Alignment
One of the most intuitive ways to prevent parameter overwriting is to use the previous iteration of the model as a teacher. Knowledge distillation acts as a regularizer that penalizes the network if its predictions or internal representations diverge too far from what it previously mastered.
For example, in dense prediction tasks like semantic segmentation, a model can easily confuse old classes with the background when those old classes are no longer explicitly labeled in new datasets. To solve this, Yang et al. introduced Label-Guided Knowledge Distillation (LGKD), which uses ground-truth labels to correctly align class probabilities between the old and new models, ensuring that the network does not overwrite the weights associated with prior classes. Similarly, in object detection, Peng et al. proposed selective and inter-related distillation to preserve the structural relationships between instances across sequential tasks.
Rather than just aligning outputs, we can also align the underlying latent space. The ICON framework introduced by Zheng et al. uses normalizing flows and KL divergence to explicitly minimize the discrepancy between partial-task and all-task aware representations, forcing the network to maintain a globally consistent latent space.
Structural Isolation: Decomposing Parameters and Changing Architectures
If sharing all parameters causes interference, an alternative strategy is to isolate them structurally. This can be done either by explicitly partitioning the network or by choosing architectures that are inherently more robust to distribution shifts.
Yoon et al. proposed Additive Parameter Decomposition (APD), a method that decomposes network weights into two distinct components: a shared base component and a sparse, task-specific component. When a new task is introduced, the model updates the shared parameters via retroactive adjustments but keeps the task-specific parameters isolated. This prevents the optimization of new objectives from destroying the unique features required for older tasks.
Interestingly, the choice of the core neural network architecture itself plays a massive role in knowledge retention. While standard Convolutional Neural Networks (CNNs) are highly susceptible to catastrophic forgetting due to their localized feature representations, Qu et al. demonstrated that Vision Transformers (ViTs) exhibit a natural robustness to sequential data shifts in federated learning environments. The self-attention mechanism in ViTs allows for more flexible, global representations that resist the destructive overwriting typical of sequential gradient updates.
Rehearsal and the Hidden Risks of Memory Buffers
When original data cannot be fully retained due to privacy or storage constraints, keeping a small, representative sample of the past—often called a coreset or memory buffer—allows the model to perform "experience replay."
To maximize the utility of a limited memory budget, the selection of these samples must be highly strategic. Nayak et al. utilized an RBF k-Determinantal Point Process (k-DPP) sampling mechanism to ensure that the memory buffer contains the most geometrically diverse set of past examples possible, preventing the model from overfitting to a narrow subset of historical data. In distributed or federated settings, Tuor et al. showed that using these coresets as proxies for offline participants prevents the global model from forgetting historical knowledge when training across rotating schedules.
However, experience replay is not a silver bullet. In a surprising theoretical and empirical revelation, Mahdaviyeh et al. demonstrated that replay can provably increase forgetting under specific task configurations. If the replayed samples are unrepresentative or introduce conflicting gradients with respect to the new task's geometry, they can inadvertently accelerate the degradation of the remaining un-rehearsed historical knowledge. This highlights that retaining knowledge is not just about keeping old data alive, but carefully managing how old and new gradients interact within the optimization landscape.