Mitigating Catastrophic Forgetting through Stability and Plasticity

This direction is merged into another direction: catastrophic forgetting (sequential learning interference). Its last version stays readable and citable here.

Version 1 of 117 papers17 conceptsfrozen 2026-10-04

Standard neural networks are remarkably fragile when learning sequences of tasks. When a model trained on Task A is subsequently trained on Task B, the gradient updates for the new objective typically overwrite the weights essential for the old one. This phenomenon, known as catastrophic forgetting, forces a choice between stability (preserving old knowledge) and plasticity (learning new information).

To solve this, researchers have moved beyond simple retraining, developing mechanisms that allow models to accumulate knowledge over time by mimicking biological memory systems.

Generative Replay and Pseudo-Rehearsal

One of the most effective ways to prevent forgetting is to "remind" the model of what it previously learned. However, storing every raw image from past tasks is often impossible due to privacy or memory constraints.

Instead of storing real data, researchers use generative replay. In this paradigm, a model learns to generate synthetic examples of the past. Christopher Kanan and Ronald Kemker proposed FearNet, which uses a dual-memory system inspired by the brain. It features a "hippocampal" generative model that creates "pseudo-examples" during a sleep-like phase to consolidate memories into a long-term "prefrontal cortex" network.

Similarly, Oleksiy Ostapenko and colleagues introduced Dynamic Generative Memory (DGM). DGM uses a conditional GAN to "hallucinate" data from previous classes. By training the current model on a mix of new real data and old synthetic data, the model maintains its decision boundaries for earlier tasks without ever seeing the original images again.

Structural Plasticity and Dynamic Expansion

If a network's capacity is fixed, learning something new must eventually come at the expense of something old. Dynamic expansion bypasses this by allowing the network architecture to grow as it encounters new tasks.

In Lifelong Learning with Dynamically Expandable Networks, Jaehong Yoon and his team developed a method (DEN) that performs "selective retraining." When a new task arrives, the model identifies which existing neurons are relevant. If the existing capacity is insufficient to learn the new task without drifting too far from the old ones, the network automatically adds new neurons. This ensures that the "semantic drift" of original weights is minimized while providing dedicated "real estate" for new knowledge.

NISPA takes this further by using sparse, rewired networks. By stabilizing critical pathways and growing new connections only where necessary, it achieves high performance with a fraction of the parameters, mimicking the brain's ability to prune and grow synaptic connections.

Synaptic Metaplasticity and Weight Protection

A third approach focuses on the weights themselves. In biological systems, not all synapses are equally plastic; some are "stiff" and resistant to change, while others are flexible.

Axel Laborieux and co-authors explored this through Synaptic Metaplasticity in Binarized Neural Networks. They use the "hidden weights" of binarized networks as a proxy for a synapse's history. If a weight has been consistently useful for past tasks, the model makes it harder to flip or change. This "intelligence" at the synaptic level allows the network to protect important knowledge without needing explicit task boundaries.

Similarly, Sen Lin proposed TRGP, which uses a "trust region" to project gradients. Instead of strictly freezing weights, TRGP learns a scaling matrix that allows new tasks to "borrow" features from old ones. This creates forward transfer—where learning Task A actually makes it easier to learn Task B—while ensuring that the updates for Task B do not interfere with the subspace occupied by Task A.

The Path Toward Brain-like Learning

The common thread across these papers is a shift from treating neural networks as static function approximators to treating them as dynamic, evolving systems. Whether through "dreaming" via generative models, expanding architecture, or modulating synaptic plasticity, the goal is to replicate the human brain's ability to learn continuously from a non-stationary stream of data.

By combining these facets—as seen in recent hybrid frameworks like BrainCL—the field is moving toward models that don't just store information, but consolidate and build upon it indefinitely.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_mitigating_catastrophic_forgetting_throu_a1d15a90,
  title = {Mitigating Catastrophic Forgetting through Stability and Plasticity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mitigating-catastrophic-forgetting-through-stability-and-plasticity/rdr_a1d15a90cd114031aed65bd99bc3e86f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_mitigating_catastrophic_forgetting_throu_a1d15a90_v1,
  title = {Mitigating Catastrophic Forgetting through Stability and Plasticity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/mitigating-catastrophic-forgetting-through-stability-and-plasticity/rdr_a1d15a90cd114031aed65bd99bc3e86f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}