When a standard neural network learns a sequence of tasks, it faces the stability-plasticity dilemma. To learn something new, the network must remain plastic, adjusting its weights to minimize error on the current task. However, this optimization process inevitably overwrites the weights optimized for previous tasks, leading to catastrophic forgetting.
Traditional approaches try to force a fixed-capacity network to compress all historical and future knowledge into the same static parameter space using weight regularization or data replay. Dynamic architecture expansion re-frames the problem entirely: instead of treating network capacity as a fixed constraint, it allows the network's structure to grow over time. By allocating new parameters for new data distributions, the model physically decouples the mechanism for retaining old knowledge from the mechanism for acquiring new skills.
The Core Mechanism: Freeze and Grow
The fundamental intuition behind dynamic expansion is simple: if you do not modify the parameters responsible for a previous task, the network cannot forget that task. As summarized in a comprehensive survey by Liyuan Wang et al., this strategy isolates task-specific knowledge into distinct structural components, limiting the scope of parameter updates during each new learning phase.
The typical expansion lifecycle follows a two-step "freeze and grow" pattern:
- Freeze: When a task is mastered, the parameters associated with it are locked.
- Grow: When a new task or data distribution arrives, the network allocates new structural components (such as neurons, layers, or sub-networks) to absorb the new information.
For instance, Fu-Yun Wang et al. demonstrate this in class-incremental learning by training independent modules for successive tasks in isolation. Because the existing parameters remain untouched, the integrity of prior knowledge is perfectly preserved. Similarly, Peiyan Zhang et al. apply this to streaming graph data, freezing established graph neural network parameters while expanding the network capacity to capture evolving structural patterns.
Fixed Architecture (Interference)
Task 1 ---> [ Shared Weights ] <--- Task 2 (Overwrites Task 1)
Dynamic Architecture (Isolation)
Task 1 ---> [ Frozen Module 1 ]
Task 2 ---> [ Newly Allocated Module 2 ] (No Interference)
Granularity of Growth: From Neurons to Experts
Architectural expansion can occur at different levels of granularity depending on the complexity of the tasks and the desired computational budget.
Coarse-Grained Expansion (Network of Experts)
At the macro level, networks expand by adding entire sub-networks or "experts" for each new task. Soochan Lee et al. utilize this approach by combining Bayesian nonparametrics with neural network experts, automatically spawning specialized sub-networks to handle new data streams. By isolating entire functional pathways, these methods ensure that complex, task-specific feature extractors do not interfere with one another.
Fine-Grained Expansion (Neuron and Channel Growth)
At the micro level, networks grow by adding individual neurons or channels within existing layers. Bo Liu et al. formulate this as a steepest descent problem in a functional neighborhood, allowing the network to decide whether to become wider or deeper during training. In a similar vein, Oleksiy Ostapenko et al. introduce a mechanism that dynamically expands layers by adding new neurons only when the available capacity of unmasked parameters falls below a specific threshold.
Parameter-Efficient Expansion (Adapters and Branches)
To avoid the high computational cost of adding full sub-networks, some approaches restrict growth to lightweight components. Shadab Khan et al. show that lifelong language learning can be managed by inserting small, task-specific adapter modules into a frozen base transformer. For computer vision tasks, Mathieu Pagé Fortin demonstrates that growing new architectural branches or detection heads on top of a shared, frozen backbone provides a physical buffer that preserves old knowledge while minimizing the parameter footprint.
Autonomous Routing and Task-Free Inference
If a network consists of multiple frozen and newly grown components, how does it know which component to use for a given input?
In task-incremental settings, the network is explicitly told which task it is performing, making routing trivial. However, in more realistic task-free settings, the network must autonomously determine when to grow and how to route data.
Novelty Detection as a Growth Trigger
To grow without explicit task boundaries, networks rely on novelty detection. Lorenzo Jamone et al. implement an architecture that autonomously spawns new local linear experts when incoming robotic exploration data significantly deviates from existing internal models. Similarly, Stefan Wermter uses an incremental growing network for robot learning that adds new nodes whenever an observed behavior falls outside the clusters of existing representations.
Gating and Routing Mechanisms
Once the architecture has expanded, inference requires routing the input to the correct specialized component. Rahaf Aljundi et al. solve this by introducing lightweight gating autoencoders alongside each expert. When a test sample arrives, each autoencoder measures its reconstruction error; the sample is then routed to the expert whose corresponding autoencoder yields the lowest error.
For large language models, Nan Du integrates expansion into a Mixture-of-Experts (MoE) framework. By adding specialized expert modules and extending the model's internal routing mechanism, the system routes tokens to the appropriate distribution-specialized experts, keeping the computational cost per token constant during inference.
Input Sample ---> [ Gating Autoencoders ] ---> Selects Best Expert ---> [ Frozen Expert ]
Managing the Efficiency Trade-off
The primary critique of dynamic architecture expansion is its scalability: if a network grows indefinitely, its parameter size and computational overhead will eventually explode. As noted by Tanvi Verma et al., expansion-based methods introduce a fundamental trade-off between absolute performance and long-term computational efficiency.
To make expansion sustainable, researchers employ two primary strategies:
- Targeted Plasticity: Instead of expanding the entire network, growth is confined to where it matters most. Dongwan Kim et al. demonstrate that restricting architectural expansion to the final blocks of a neural network focuses plasticity on high-level feature formation, drastically reducing the computational cost of adding full sub-networks while maintaining high accuracy.
- Sparsity and Pruning: Networks can optimize their existing space before triggering growth. Sangwon Jung et al. propose a hybrid approach that combines parameter isolation with group sparse regularization. By exactly freezing important nodes and pruning unimportant ones, the model ensures that it scales its physical capacity only after its internal nodes are fully utilized.
By decoupling stability and plasticity through structural growth, dynamic architecture expansion provides a mathematical guarantee against forgetting, shifting the challenge of continual learning from optimization constraints to intelligent resource allocation.