The Stability-Plasticity Dilemma in Continual Learning

This direction is split into other directions: regularization-induced rigidity. Its last version stays readable and citable here.

Version 1 of 1174 papers199 conceptsfrozen 2026-10-04

When a neural network learns a new task, it typically updates its weights to minimize error on the new data. However, in a sequential learning setting, these updates often overwrite the parameters essential for previous tasks, leading to a total loss of prior knowledge—a phenomenon known as catastrophic forgetting.

To solve this, researchers face the stability-plasticity dilemma: a model must be "plastic" enough to acquire new information but "stable" enough to prevent the interference of new gradients with old memories.

The Cost of Stability: Model Intransigence

The most common way to enforce stability is through regularization. Techniques like Elastic Weight Consolidation (EWC), explored by Ben Glocker and Konstantinos Kamnitsas in medical imaging, identify which weights are "important" for past tasks and penalize changes to them.

However, this creates a secondary problem: model intransigence. As Denis Jered McInerney and Byron C Wallace observe, if regularization is too aggressive, the network becomes too "stiff" to learn anything new. The model is so focused on staying close to its previous state that it cannot adapt to the unique characteristics of the current task. This tension is particularly visible in generative models, where Kamil Deja and Tomasz Trzcinski found that simple weight penalties often lead to a collapse in the quality of generated outputs.

Granular Control: Beyond Global Penalties

To move past the limitations of global weight penalties, researchers have developed more surgical methods of control:

  • Neuron-Level Control: Instead of looking at connection weights, Inyoung Paik and Injung Kim 0001 proposed Neuron-level Plasticity Control (NPC). By dynamically adjusting the learning rates of individual neurons based on their importance to past tasks, they achieve a more efficient balance than traditional weight consolidation.
  • Sparse Allocation: Rather than using the whole network for every task, Yijun Yang and Tianyi Zhou developed CoTASP, which uses sparse "prompts" or masks to carve out specific sub-networks for each task. This allows similar tasks to share knowledge while physically isolating the parameters of distinct tasks to prevent overwriting.
  • Structural Expansion: Jaehong Yoon and Sung Ju Hwang introduced Dynamically Expandable Networks (DEN), which automatically decide when to retrain existing neurons and when to "split" or add new ones to accommodate new information without drifting away from old concepts.

The Pre-training Paradigm Shift

A major realization in recent years is that the "best" continual learning algorithms change depending on where the model starts. Yu-Xiong Wang and Kuan-Ying Lee demonstrated that starting with Pre-trained Models (PTMs) completely reshuffles the leaderboard of algorithms.

In PTMs, the challenge isn't just preventing forgetting; it's leveraging the rich, existing feature representations. James Seale Smith and Junjiao Tian showed that for PTMs, even simple L2 regularization can outperform complex state-of-the-art methods because the model already possesses the necessary "vocabulary" to understand new tasks. The goal shifts from learning from scratch to adapting without destroying.

The Risk of Entrenchment

Stability has a hidden danger: it can protect bad habits. Kai Gu introduced the Einstellung Rigidity Index (ERI) to diagnose when a model is "stuck." He found that mechanisms designed to prevent forgetting—like replay buffers or strict regularization—can inadvertently entrench suboptimal "shortcuts" or features learned early in training. This rigidity prevents the model from discarding irrelevant information, effectively blocking it from learning more robust, task-specific features later on.

Ultimately, the field is moving away from seeing stability and plasticity as a zero-sum game. Instead, through methods like Information Maximization (Lama Alssum) and Mutual Information Maximization (Ziqi Gu), researchers are seeking objectives that guide the model to remain consistent with its past while remaining flexible enough to find optimal solutions for the future.

Papers in this version

Showing 60 of 174 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_the_stability_plasticity_dilemma_in_cont_037d702a,
  title = {The Stability-Plasticity Dilemma in Continual Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-stability-plasticity-dilemma-in-continual-learning/rdr_037d702af5f34efcadb264ff2671665c},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_the_stability_plasticity_dilemma_in_cont_037d702a_v1,
  title = {The Stability-Plasticity Dilemma in Continual Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-stability-plasticity-dilemma-in-continual-learning/rdr_037d702af5f34efcadb264ff2671665c/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}