Linear Parameter Growth in Modular Continual Learning

152 concepts133 papers2 proposals

When training a neural network on a continuous stream of tasks, deep learning models face a fundamental dilemma: stability versus plasticity. If you update the entire network to learn a new task, it will inevitably overwrite previously acquired knowledge—a phenomenon known as catastrophic forgetting.

To bypass this problem, a popular and highly effective strategy is parameter isolation. Instead of modifying the core network, researchers freeze a pre-trained base model and allocate lightweight, task-specific modules (such as adapters, prompts, or calibration layers) for each incoming task. Because the parameters for older tasks are locked, forgetting is structurally impossible.

For instance, Pravendra Singh et al. demonstrate in Calibrating CNNs for Lifelong Learning that adding tiny task-specific calibration modules can preserve past knowledge with negligible initial overhead. Similarly, Yunhao Ge et al. introduce CLR: Channel-wise Lightweight Reprogramming for Continual Learning, appending micro-layers to a frozen backbone to achieve zero forgetting across dozens of datasets.

At first glance, this feels like a free lunch. However, as these systems scale to long-term scenarios, they encounter a hidden tax: linear parameter growth.

The Linear Scaling Tax

The core illusion of modular continual learning is that because an individual task module is small, the overall system remains efficient. In reality, if a system instantiates a new set of parameters for every single task, the total parameter count scales linearly ($O(N)$) with the number of tasks encountered. As Yulai Cong et al. observe in GAN Memory with No Forgetting, this lack of sub-linear scaling creates a severe scalability bottleneck for systems intended to operate over hundreds or thousands of sequential learning stages.

This linear accumulation manifests across nearly every flavor of modular architecture:

Beyond Storage: The Routing and Inference Bottleneck

The linear parameter growth problem is not confined to disk storage; it actively degrades inference efficiency and algorithmic complexity.

When a modular model receives an input at inference time, it faces a routing problem: it must determine which specialized module to activate. To do this, models often maintain task locators, routers, or distribution detectors. Ziqin Wang et al. point out in CoPE: Continual Probe-guided Expansion for Large Vision-Language Models that maintaining a dedicated set of these routing modules for every learned task causes the routing bank itself to grow linearly, introducing significant computational overhead as the task sequence lengthens.

Input Data ───► [ Routing Bank (Grows O(N)) ] 
                       │
         ┌─────────────┼─────────────┐
         ▼             ▼             ▼
    [Module 1]    [Module 2]    [Module N]  ◄── (Total Modules Grow O(N))

In worse-case scenarios, the interaction between these modules causes computational complexity to scale super-linearly:

  • Quadratic Latency: If a modular network attempts to compose or aggregate the internal representations of all previous modules to solve a new task, the computational graph expands rapidly. Mikel Malagon et al. analyze this behavior in Self-Composing Policies for Scalable Continual Reinforcement Learning, demonstrating that while composing only the final outputs of modules preserves linear growth, densely connecting the internal hidden layers of all historical modules causes inference complexity to explode quadratically.
  • Cubic Complexity: In non-parametric Bayesian formulations, the scaling can be even more severe. Sanyam Kapoor et al. show in Variational Auto-Regressive Gaussian Processes for Continual Learning that allocating new inducing points for every sequential task can cause inference complexity to scale cubically with the number of tasks, rendering long-term deployment impossible without explicit information distillation.

The Knowledge Sharing Blockade

Beyond the physical constraints of memory and compute, naive linear parameter growth introduces a conceptual flaw: it treats every task as an isolated island.

True lifelong learning implies that a model should become a better learner over time by transferring knowledge from past experiences to accelerate the acquisition of new ones. When an architecture blindly instantiates a brand-new module for every task boundary, it fails to recognize structural commonalities between related domains.

As argued by Haodong Lu et al. in Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learning, a strict one-to-one mapping between tasks and modules prevents the model from identifying and exploiting shared features. If Task 100 is 95% identical to Task 5, a linear expansion strategy will still allocate a redundant, independent module for Task 100, wasting capacity and missing an opportunity for forward knowledge transfer.

Breaking the Linear Bottleneck

To transition from linear parameter scaling to sustainable, sub-linear scaling, recent research focuses on breaking the rigid one-to-one mapping between tasks and parameters.

One approach is conditional self-expansion. Instead of adding a module automatically at every task boundary, models use distribution descriptors to evaluate whether the incoming data is genuinely novel. Haodong Lu et al. utilize this strategy to allow architectures to expand their capacity only when significant distribution shifts occur, achieving sub-linear growth and encouraging module reuse across related tasks.

Another prominent direction is post-hoc consolidation and distillation. Rather than allowing the library of modules to grow indefinitely, models periodically merge or distill redundant components. For example, Tejas Srinivasan et al. introduce I2I: Initializing Adapters with Improvised Knowledge, which "improvises" solutions by combining existing modules and distilling that knowledge into compact representations, pointing the way toward a fixed or sub-linear memory footprint.

Similarly, Yaoyue Zheng demonstrates in Revisiting Weight Regularization for Low-Rank Continual Learning that by estimating parameter importance over a shared low-rank space rather than isolating tasks into discrete modules, models can achieve a stable stability-plasticity trade-off with a memory footprint that remains entirely constant, regardless of how many tasks are added.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_linear_parameter_scaling_linear_paramete_53a4eeaf,
  title = {linear parameter scaling (linear parameter growth)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-parameter-scaling-linear-parameter-growth/rdr_53a4eeafe37645039ea945845a5fbac9},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_linear_parameter_scaling_linear_paramete_53a4eeaf_v1,
  title = {linear parameter scaling (linear parameter growth)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/linear-parameter-scaling-linear-parameter-growth/rdr_53a4eeafe37645039ea945845a5fbac9/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake