When a neural network learns a sequence of tasks over time, it faces a fundamental challenge known as catastrophic forgetting: optimizing weights for a new task typically overwrites the representations learned for previous ones.
To bypass this problem, researchers often turn to architecture-based continual learning. Instead of forcing a single set of weights to handle every distribution, these methods isolate parameters or dynamically add new modules—such as task-specific adapters, sub-networks, or architectural branches—for each incoming task. By freezing old parameters and routing data through dedicated pathways, interference between tasks drops to zero.
However, this strategy introduces a severe scalability paradox. If a model allocates a fixed set of new parameters for every novel domain or class it encounters, its memory footprint grows linearly with the number of tasks. In open-ended or long-term streaming scenarios, this leads to an unsustainable parameter explosion, making deployment on resource-constrained hardware or edge devices impossible.
To build truly lifelong learning systems, researchers must find ways to expand a model's capacity without triggering unbounded architectural growth.
The Scalability Paradox of Network Expansion
The intuition behind expanding a network is straightforward: if you run out of room in your current representation space, build a new branch. For example, in agricultural robotics, Mathieu Pagé Fortin demonstrates in Class-Incremental Learning of Plant and Disease Detection: Growing Branches with Knowledge Distillation that growing new architectural branches effectively mitigates forgetting when detecting new plant diseases.
Yet, as highlighted by toolboxes like PyCIL developed by Da-Wei Zhou in PyCIL: a Python toolbox for class-incremental learning, this cumulative growth creates a massive computational bottleneck. Over a long operational lifetime, a model can experience an unsustainable "memory explosion." Weicheng Meng et al. note in PCLR: Progressively Compressed LoRA for Multimodal Continual Instruction Tuning that adding uncompressed modules for every sequential task quickly makes models too large for practical storage.
The core challenge is that naive expansion treats every task as completely independent, ignoring the structural overlap between them. To prevent parameter explosion, modern architectures must transition from unbounded linear growth to parameter-efficient adaptation.
Low-Rank Subspaces and Parameter-Efficient Adaptation
One of the most effective ways to curb parameter growth is to restrict task-specific updates to a low-rank subspace. Because the intrinsic dimensionality of adapting to a new task is often much lower than the full weight space of a network, we do not need to add full-sized layers to capture new knowledge.
Instead of expanding the network with heavy, high-dimensional branches, models can inject lightweight, low-rank increments:
- Low-Rank Adaptation (LoRA): Vince Zhu et al. apply this principle to medical imaging in Low-Rank Continual Pyramid Vision Transformer: Incrementally Segment Whole-Body Organs in CT with Light-Weighted Adaptation. By freezing a base Vision Transformer and adding low-rank adaptation layers, they achieve high accuracy while using 83% fewer new parameters than traditional expansion methods. Similarly, Rakib Hyder et al. in Incremental Task Learning with Incremental Rank Updates eliminate forgetting by freezing past task factors and appending only rank-1 updates for new tasks.
- Orthogonal Subspace Projections: Rather than adding parameters, some methods constrain weight updates to remain entirely orthogonal to the subspaces critical for past tasks. Ligong Han et al. introduce this in Sculpting Subspaces: Constrained Full Fine-Tuning in LLMs for Continual Learning via Orthogonal Subspace Fine-Tuning (OSFT). By using Singular Value Decomposition (SVD) to identify and protect critical weight directions, they allow full fine-tuning within a strictly fixed parameter footprint.
Naive Expansion:
Task 1: [Base Model] + [Full Layer 1]
Task 2: [Base Model] + [Full Layer 1] + [Full Layer 2] --> Linear Parameter Growth
Low-Rank / Subspace Adaptation:
Task 1: [Base Model] + [Low-Rank Adapter 1]
Task 2: [Base Model] + [Low-Rank Adapter 1] + [Low-Rank Adapter 2] --> Minimal Footprint Growth
Capacity Recycling via Pruning and Masking
An alternative to adding external parameters is to exploit the inherent overparameterization of deep neural networks. Instead of expanding the physical boundaries of the network, these approaches treat the existing architecture as a finite resource, finding and "recycling" underutilized capacity.
[ Task 1 Masked Weights ] [ Unused Free Space ] [ Task 2 Masked Weights ]
└───────────┬───────────┘ └──────────┬─────────┘ └───────────┬───────────┘
Frozen for Task 1 Allocated for Task 2 Frozen for Task 3
In SpaceNet: Make Free Space for Continual Learning, Ghada Sokar et al. demonstrate how adaptive sparse training can pack multiple tasks into a fixed-capacity network. By compressing each task into a compact set of neurons and reserving them via masks, they prevent forgetting without any structural expansion.
When expansion is necessary, it can be paired with aggressive pruning to keep the model parsimonious. Zifeng Wang et al. propose the Learn-Prune-Share for Lifelong Learning (LPS) framework, which uses Alternating Direction Method of Multipliers (ADMM) pruning to partition a single network into task-specific pieces, ensuring that parameters are actively shared and reused rather than duplicated.
Similarly, Binzong Geng et al. develop the TPEM framework in Continual Learning for Task-oriented Dialogue System with Iterative Network Pruning, Expanding and Masking, which dynamically expands capacity only when existing unmasked neurons are insufficient to absorb the new domain's complexity.
Compositional Knowledge Sharing
The most sustainable way to scale an architecture is to allow new tasks to reuse pieces of old tasks. If Task 3 is a combination of features from Task 1 and Task 2, the model should compose existing experts rather than spawning a new one.
Bin Wu et al. implement this compositional philosophy in Adaptive Compositional Continual Meta-Learning (ACML). They associate tasks with a composition of shared meta-knowledge components using an Indian Buffet Process prior. This allows heterogeneous tasks to share underlying parameters and adaptively prune redundant components, preventing the infinite growth common in modular systems.
In the context of Vision Transformers, Chinmay Savadikar et al. introduce CHEEM in Continual Learning via Learning a Continual Memory in Vision Transformer. CHEEM employs an efficient search mechanism at the attention projection layers to explicitly decide whether to reuse an existing module, adapt it, or create a new one. This prevents redundant parameter growth by exploiting structural synergies between similar tasks.
When new experts must be added, frameworks like Dense Network Expansion (DNE), developed by Dashan Gao et al. in Dense Network Expansion for Class Incremental Learning, use a "Task Attention Block" to let new modules selectively read and reuse features from older branches, keeping the size of the newly added experts highly compact.
Finally, to handle completely open-ended streams where task boundaries are unknown, Fei Ye proposes in Task-Free Dynamic Sparse Vision Transformer for Continual Learning an energy-based novelty detection mechanism. The architecture expands dynamically only when a significant spike in data novelty is detected, balancing the stability of frozen historical parameters with the targeted allocation of new, sparse capacity.