Modular Parameter Isolation for Continual Learning

This direction is split into other directions: parameter isolation. Its last version stays readable and citable here.

Version 1 of 1897 papers1,207 conceptsfrozen 2026-10-04

Standard neural networks suffer from a fundamental flaw known as catastrophic forgetting: when trained on a new task, the gradients necessary to learn new patterns overwrite the weights that stored previous knowledge. This "stability-plasticity dilemma" forces a choice between a model that is stuck in the past or one that has no memory of it.

Researchers have converged on a powerful solution: Modular Parameter Isolation. Instead of treating a model as a monolithic block of weights, this approach treats it as a collection of specialized, often independent units. By isolating the parameters used for different tasks, models can acquire new skills with surgical precision, leaving existing knowledge untouched.

The Physics of Forgetting

To understand why isolation works, we must look at how a model learns. In a standard Transformer or CNN, every parameter is potentially "up for grabs" during backpropagation. When Andrei A. Rusu and colleagues introduced Progressive Neural Networks, they proposed a physical barrier to this process. By instantiating a new neural "column" for every new task and freezing the weights of all previous columns, they ensured that the gradient updates for Task B could never physically alter the weights of Task A.

This mechanism works because it transforms the learning objective from "reconfigure the whole brain" to "add a new specialized circuit." As William Whitney 0001 demonstrated in Disentangled Representations in Neural Models, routing specific tasks to isolated functional units ensures that the weights associated with inactive modules remain preserved.

Adapters and Experts as Knowledge Buffers

Modern approaches have moved away from adding entire "columns" (which is computationally expensive) toward more efficient "parameter-efficient" modules.

  1. Task-Specific Adapters: In Auto-selected Knowledge Adapters for Lifelong Person Re-identification, Ruiqi Wu uses low-rank adaptation (LoRA) units. The base model remains frozen—acting as a permanent library of general features—while tiny, task-specific modules are plugged in to handle new domains.
  2. Mixture of Experts (MoE): Models like Self-MoE by Junmo Kang and LEMoE by Renzhi Wang transform the LLM into a system of specialized experts. When new knowledge is encountered, the system allocates a new "expert" module. Because the router learns to send specific inputs only to the relevant expert, the "generalist" knowledge in the rest of the model is never corrupted.

This modularity is even effective at the class level. Michał Zając and Gido M. van de Ven proposed Prediction Error-based Classification (PEC), which assigns a dedicated "student" network to every single class. By freezing these students after they learn their specific category, the model achieves perfect recall of old classes even as it learns thousands of new ones.

Finding the Hidden Modules

Isolation doesn't always require adding new parameters; sometimes it involves finding the "dead space" or specific "skill zones" within an existing model.

  • Sparse Sub-networks: Jianze Liang and Lei Li developed PRUNE-TUNE, which uses pruning to discover sparse, domain-specific sub-networks. By masking which weights are allowed to change for a specific domain, they prevent interference between, for example, medical translation and legal translation.
  • Selective Updates: Wenxuan Zhang showed in Selective Parameter Update (SPU) that updating as little as 3% of a model's parameters—specifically those most sensitive to the new task—can prevent the loss of pre-trained knowledge. This suggests that "modularity" can be an emergent property of how we apply gradients, not just a structural choice.

The Trade-off: Stability vs. Transfer

The primary reason modular isolation works so well is that it prioritizes stability. By design, if you freeze the weights of Task A, the performance on Task A cannot drop.

However, as noted by Michal Zajac 0005, this creates a fundamental trade-off: the total elimination of forgetting often comes at the expense of forward knowledge transfer. If Task A and Task B are completely isolated, the model cannot use the "logic" it learned in Task A to help it learn Task B faster.

To bridge this, researchers like Andrei A. Rusu use lateral connections between modules, and Tim Klinger in Compositional Program Generation uses formal grammar to link independent neural modules. These links allow the model to compose existing skills into new behaviors without ever needing to modify the underlying "skill" parameters themselves.

Papers in this version

Showing 60 of 897 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_modular_parameter_isolation_for_continua_2e23fa54,
  title = {Modular Parameter Isolation for Continual Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/modular-parameter-isolation-for-continual-learning/rdr_2e23fa546fc340d8b9b87da1c0689b5d},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_modular_parameter_isolation_for_continua_2e23fa54_v1,
  title = {Modular Parameter Isolation for Continual Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/modular-parameter-isolation-for-continual-learning/rdr_2e23fa546fc340d8b9b87da1c0689b5d/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}