When a human learns to ride a bicycle, they do not suddenly forget how to walk. However, artificial neural networks have historically struggled with this basic requirement of intelligence. In a standard training setup, if you take a model proficient at Task A and train it on Task B, its performance on Task A typically collapses. This phenomenon is known as catastrophic forgetting (or catastrophic interference).
The root of the problem lies in the way gradient descent updates weights. In a dense neural network, every weight potentially contributes to every output. When the network learns a new task, the global weight updates required to minimize the new loss function effectively overwrite the specific configurations that encoded the previous task. As Ian Goodfellow and Yoshua Bengio noted in An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networks, the optimization process for new data is essentially "blind" to the requirements of the old data.
To solve this, researchers must navigate the stability-plasticity dilemma: the network must be "plastic" enough to acquire new knowledge but "stable" enough to prevent the interference of that new knowledge with existing memories.
Regularization: Protecting Important Weights
One intuitive approach is to identify which weights are most "important" for previous tasks and discourage the network from changing them.
In the seminal paper Overcoming catastrophic forgetting in neural networks, James Kirkpatrick and colleagues at DeepMind introduced Elastic Weight Consolidation (EWC). EWC uses the Fisher Information Matrix to estimate the importance of each parameter. When training on a new task, a penalty term is added to the loss function that scales with the importance of the weight to previous tasks. This "stiffens" the critical connections, forcing the network to find solutions for the new task in the "free" dimensions of the weight space.
Other researchers have framed this as a Bayesian problem. In Variational Continual Learning, Cuong V. Nguyen and Richard E. Turner treat the posterior distribution of weights from a previous task as the prior for the next. Similarly, Chen Zeno and Elad Hoffer proposed Bayesian Gradient Descent (BGD) in Task Agnostic Continual Learning Using Online Variational Bayes, which uses weight uncertainty to naturally scale learning rates, preserving "certain" (important) knowledge.
Architectural Isolation: Creating Dedicated Space
If the problem is that tasks overwrite each other's weights, a direct solution is to give each task its own dedicated parameters or "sub-network."
Arun Mallya explored this through parameter redundancy. In PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning, the model identifies unimportant weights for Task A, prunes them, and then freezes the remaining "important" weights. These frozen weights are then used as a fixed backbone for Task B, which only trains on the previously pruned (now free) weights. A related approach, Piggyback, learns task-specific binary masks that "hide" or "reveal" different parts of a fixed, pre-trained network for different tasks.
For more complex scenarios, Rahaf Aljundi and Tinne Tuytelaars proposed Expert Gate, which builds a growing collection of specialized expert models. A gating mechanism (an autoencoder) determines which expert is best suited for a given input, ensuring that the model for Task A is never even touched when processing Task B.
Rehearsal and Replay: Refreshing the Memory
Biological brains often consolidate memories during sleep by "replaying" neural patterns. Continual learning algorithms mimic this by revisiting old data while learning new tasks.
However, storing all previous raw data is often impossible due to memory constraints or privacy concerns. Michiel van der Ven argues in Generative replay with feedback connections as a general strategy for continual learning that Generative Replay is the most robust solution. Instead of storing data, the system trains a generative model (like a VAE or GAN) to "dream" up synthetic examples of previous tasks. These synthetic examples are then interleaved with new data during training, effectively reminding the network of what it already knows.
For rapid adaptation, Pablo Sprechmann and Jack W. Rae introduced Memory-based Parameter Adaptation (MbPA) in Memory-based Parameter Adaptation. This method stores specific experiences in an episodic memory. At test time, the model performs a very fast, local weight update based on similar past experiences, allowing it to handle new information without permanently altering its long-term weights.
The Path Forward
The field is moving toward more efficient, task-agnostic systems. Recent work like POP (Prompt Of Prompts) by Dashan Gao leverages frozen foundation models, using small, learnable "prompts" to navigate task-specific knowledge. By shifting the focus from changing the entire network to managing how we interface with it, researchers are bringing us closer to models that can learn continuously throughout their "lifetimes" just as humans do.