When a human learns to ride a bicycle, they don’t suddenly forget how to walk. In contrast, standard artificial neural networks suffer from catastrophic forgetting: when trained on a new task, the gradient updates required to master the new data often overwrite the weights essential for previous tasks.
This tension is known as the stability-plasticity dilemma. A model must be plastic enough to integrate new information but stable enough to preserve existing representations. Research across 134 papers suggests that solving this is not about finding a single "trick," but about managing how knowledge is stored, protected, and occasionally discarded.
Regularization as a Constraint on Change
The most direct way to ensure stability is to penalize the network for changing weights that were important for previous tasks. This is the core intuition behind regularization-based methods.
Sungmin Cha and colleagues introduced CPR: Classifier-Projection Regularization for Continual Learning, which acts as a "patch" to keep parameter changes within a safe subspace. Similarly, Sinan Özgür Özgün demonstrated in Importance Driven Continual Learning for Segmentation Across Domains that we can dynamically adjust learning rates for specific parameters. If a weight is critical for a previous brain MRI segmentation task, its learning rate is lowered during the next task to prevent erosion.
From a Bayesian perspective, this is equivalent to treating the posterior distribution of the previous task as the prior for the current one. Cuong V. Nguyen formalized this in Variational Continual Learning, providing a principled framework where the network "remembers" by narrowing the range of acceptable weight values as it sees more tasks.
Memory Rehearsal and Generative Recall
If regularization is about "not changing," rehearsal is about "reminding." By keeping a small buffer of old data (a coreset) and interleaving it with new data, the model maintains a constant signal of previous knowledge.
However, storing raw data is often impossible due to privacy or memory constraints. Weiming Dong and his team proposed Incremental Concept Learning via Online Generative Memory Recall, which uses a Generative Adversarial Network (GAN) to "recall" pseudo-samples of old concepts. Instead of storing pixels, the model learns to dream up old tasks, using these dreams to stabilize its feature representations during new learning.
Architectural Decoupling and Subspace Adaptation
A more structural approach to the dilemma is to separate the "stable" parts of a network from the "plastic" ones. By isolating task-specific updates, we can prevent interference entirely.
Yuying Hao implemented this in RAIS: Robust and Accurate Interactive Segmentation via Continual Learning by splitting the architecture into a fixed base module and a flexible adaptation module. This ensures that the foundational features remain untouched while the model learns the nuances of a new domain.
In the era of large pre-trained models, this often takes the form of parameter-efficient fine-tuning. Huancheng Chen developed Dual Low-Rank Adaptation (DualLoRA), which uses two parallel adapters—one to preserve old knowledge and one to acquire new skills. Similarly, Didi Zhu showed in Model Tailor that we can identify a sparse "model patch" (less than 10% of parameters) to update, leaving the rest of the multi-modal LLM stable.
The Surprising Necessity of Forgetting
While most research focuses on preventing forgetting, some evidence suggests that "active forgetting" is actually a requirement for efficient learning. If a model tries to remember everything with equal intensity, it becomes too rigid to adapt to genuinely new or conflicting data.
Liyuan Wang took inspiration from the fruit fly brain to develop Collaborative Continual Learners with Active Forgetting (CAF). By intentionally pruning irrelevant or conflicting information, the system achieves a better balance between memory and adaptability.
This is echoed by Ben Sanati in Forgetting is Everywhere, who argues that forgetting is not a bug but a feature of adaptive systems. By defining forgetting as a lack of predictive self-consistency, they show that approximate learners (like neural networks) actually benefit from a moderate amount of forgetting to adapt quickly to non-stationary environments.