Modular Parameter Isolation for Continual Learning

446 concepts325 papers1 proposal

When a standard neural network learns a sequence of tasks, gradient descent updates its weights globally. This holistic update causes catastrophic forgetting: representations optimized for an earlier task are overwritten to accommodate new data distributions.

Modular parameter isolation solves this problem by drawing physical boundaries between tasks within the model's architecture. Instead of modifying the entire network, these methods freeze existing parameters and route new information to dedicated, isolated parameter subsets. By preventing gradients from backpropagating through previously learned weights, modular isolation mathematically guarantees that historical knowledge remains perfectly intact.

The Anatomy of Shared and Private Parameters

The core intuition behind parameter isolation is that knowledge can be factorized into two categories: domain-shared features (like edges in vision or syntax in language) and domain-specific features (like unique object classes or specialized jargon).

To exploit this, architectures are typically split into a frozen shared backbone and dynamic, private blocks. For example, in neural machine translation, Yongchao Deng et al. developed the Factorized Transformer for Multi-Domain Neural Machine Translation, which decouples general language patterns from domain-specific blocks. This architectural decoupling ensures that optimizing for a new domain does not degrade performance on previously learned ones, a benefit also demonstrated by Lixin Su et al. in Continual Domain Adaptation for Machine Reading Comprehension.

Standard Fine-Tuning (Interference)
[Input] ──> [ Shared & Overwritten Weights ] ──> [Output]

Modular Isolation (Preservation)
            ┌──> [ Frozen Base Backbone ] ──┐
[Input] ────┼──> [ Isolated Task Module 1 ] ┼──> [Output]
            └──> [ Isolated Task Module 2 ] ┘

By keeping the core model static, the system can scale to accommodate numerous new data sources. As shown by Sylvestre-Alvise Rebuffi et al. in Learning multiple visual domains with residual adapters, appending tiny, task-specific residual layers to a frozen feature extractor allows a single compact representation to solve vastly different visual tasks without cross-task interference.

Mechanisms of Isolation: Adapters, Prompts, and Low-Rank Updates

To prevent parameter explosion as the number of sequential tasks grows, researchers employ parameter-efficient design patterns to isolate new knowledge.

Low-Rank Increments

Instead of adding large blocks of layers, models can isolate task-specific knowledge in external, compact matrices. Bohao PENG et al. introduced this in the Scalable Language Model with Generalized Continual Learning, where low-rank weight increments are added to a frozen base model. This strategy is highly effective for scaling retrieval systems; Yuxiang Guo et al. demonstrated in Birdie: Natural Language-Driven Table Discovery Using Differentiable Search Index that confining new knowledge to low-rank adaptation modules reduces catastrophic forgetting by over 90% in dynamic indexing environments.

Prompt Tuning

Rather than modifying internal network weights, knowledge can be isolated within the input space. In MoP-CLIP: A Mixture of Prompt-Tuned CLIP Models for Domain Incremental Learning, Julien Nicolas et al. showed that learning unique sets of visual and text prompts for each new data distribution—while keeping the underlying foundation model frozen—safeguards historical capabilities.

Feature Map Transformations

Modules can also operate directly on intermediate activations. In CAM-GAN: Continual Adaptation Modules for Generative Adversarial Networks, Sakshi Varshney et al. appended lightweight task-specific modules directly to feature maps, isolating new task knowledge without altering the base generative parameters.

The Routing Challenge: Activating the Right Expert

If knowledge is isolated into distinct modules, how does the model know which module to activate at inference time? This is known as the routing problem.

When a task identity label is explicitly provided, the model can simply activate the corresponding sub-network. However, in real-world scenarios where the task ID is unknown, the system must dynamically route inputs to the correct expert.

[ Inference Input ] ──> [ Router / Gating Network ]
                              │
               ┌──────────────┴──────────────┐
               ▼                             ▼
     [ Isolated Expert 1 ]         [ Isolated Expert 2 ]

To solve this, Vladimir Araujo et al. proposed a framework in Learning to Route for Dynamic Adapter Composition in Continual Learning with Language Models that trains modular adapters in total isolation, preventing feature drift, and then uses a small memory buffer to train a specialized router that dynamically composes them at inference time.

Similarly, Tianlun Zheng et al. designed the MRN: Multiplexed Routing Network for Incremental Multilingual Text Recognition, which uses a router to weigh predictions from language-specific experts. To avoid relying on historical data buffers for routing, Sanket Vaibhav Mehta et al. developed Generate to Discriminate: Expert Routing for Continual Learning. Their approach uses synthetic data to train a router that directs test samples to domain-specific experts, ensuring that updates for new tasks never cause parameter conflicts in a shared space.

Granular Isolation: Class-Specific Modules

The principle of isolation can be pushed to its logical extreme: allocating distinct modules not just for different tasks or domains, but for every single class.

In Prediction Error-based Classification for Class-Incremental Learning, Michał Zając proposed training class-specific modules to mimic a frozen random network. Because the parameters for previous classes are never updated during the training of new ones, the model maintains perfect recall of earlier categories without data rehearsal.

A common manifestation of this concept is decoupling feature extraction from classification. For instance, Eden Belouadah et al. introduced DeeSIL: Deep-Shallow Incremental Learning, which pairs a fixed deep feature extractor with independent, fast-to-train shallow SVMs for each new class. Because the learning of new categories does not modify the parameters associated with previously learned categories, catastrophic forgetting is structurally eliminated.

Following a similar logic, Juan-Manuel Pérez-Rúa et al. designed an incremental object detector in Incremental Few-Shot Object Detection by splitting the architecture into a frozen, class-generic feature extractor and a dynamic object locator that treats new categories as independent modular components.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_modular_parameter_isolation_cc30e80e,
  title = {modular parameter isolation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/modular-parameter-isolation/rdr_cc30e80edf59463bae025e6ce80e65f1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_modular_parameter_isolation_cc30e80e_v1,
  title = {modular parameter isolation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/modular-parameter-isolation/rdr_cc30e80edf59463bae025e6ce80e65f1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake