When a neural network learns a sequence of tasks over time, it faces a fundamental tension known as the stability-plasticity dilemma. As outlined by David Isele in Representations for Continuous Learning, plasticity is the network's capacity to integrate new data distributions, while stability is its ability to safeguard historical knowledge.
Standard gradient descent optimizes globally for the current task, which routinely overwrites the parameter configurations optimized for previous tasks—a phenomenon called catastrophic forgetting. Conversely, if a network is made too stable, it becomes rigid, failing to accumulate new knowledge or adapt to shifting environments, a limitation highlighted by Andrea Cossu et al. in Continual Learning for Human State Monitoring.
To resolve this dilemma, modern continual learning research focuses on understanding and manipulating the geometry, modularity, and capacity of the model's representation space.
The Geometry of the Parameter Space
To understand why models forget, we can view continual learning as a trajectory through a loss landscape. Supriyo Chakraborty et al. formalize this process as a sequence of optimal control problems in On Understanding of the Dynamics of Model Capacity in Continual Learning. When a model optimizes for a new task, the gradient steps push the parameters out of the low-loss regions of previous tasks.
One intuitive solution is to shape the loss landscape to be more forgiving. Iman Mirzadeh et al. demonstrate in Understanding the Role of Training Regimes in Continual Learning that specific training regimes—such as utilizing dropout and smaller batch sizes—encourage the model to converge to wider local minima. A wider minimum means that minor parameter shifts during subsequent tasks will not drastically increase the loss on older tasks, inherently boosting stability.
Standard Regime (Narrow Minimum):
Task 1 Loss: \___/ --> Small parameter shift moves model completely out of the well.
Stable Regime (Wide Minimum):
Task 1 Loss: \_______/ --> Small parameter shift keeps model inside the low-loss region.
Alternatively, we can actively constrain the direction of the gradient updates. In Restricted Orthogonal Gradient Projection for Continual Learning, Zeyuan Yang et al. explore projecting gradients into subspaces that do not interfere with past tasks. While strict orthogonal projection prevents forgetting, it can block forward transfer (the ability of past knowledge to aid new learning). The authors resolve this by allowing "oblique" gradient updates that balance retention with the flexibility needed to learn correlated tasks. This concept is mirrored in matrix-driven approaches like CODE-CL: Conceptor-Based Gradient Projection for Deep Continual Learning, where gradients are adaptively projected based on the degree of task similarity.
Modular Isolation and Parameter Routing
Instead of forcing the entire network to handle every task, another strategy isolates parameters. If different weights are responsible for different tasks, updates for a new task cannot overwrite the knowledge of a previous one.
This can be achieved via parameter-efficient tuning and modular architectures. For instance, in ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptation, Jiaming Liu et al. propose a dual-path adapter architecture. A low-rank adapter maintains domain-shared, stable features, while a high-rank adapter captures domain-specific, plastic shifts. Similarly, Yuhang He introduces Beyond Prompt Learning: Continual Adapter for Efficient Rehearsal-Free Continual Learning, which utilizes parameter-expandable adapter layers to isolate task-specific weights, eliminating the structural interference that degrades performance on previous tasks.
Isolation can also happen at the level of normalization statistics. In Task-Specific Normalization for Continual Learning of Blind Image Quality Models, Xiaokang Yang 0001 et al. freeze the core feature extractor entirely to guarantee stability, while learning task-specific normalization parameters to provide the necessary plasticity. This directly prevents the "recency bias" of normalization statistics analyzed by Yilin Lyu in Overcoming Recency Bias of Normalization Statistics in Continual Learning: Balance and Adaptation, where running statistics quickly forget old tasks to favor the current data distribution.
Dual-Teacher Distillation: Anchoring the Representation Space
When a model must adapt its representations without changing its architecture or expanding its parameters, knowledge distillation acts as an anchor. However, distilling solely from the immediate past model can lead to a slow drift away from foundational knowledge over a long sequence of tasks.
To solve this, researchers use dual-teacher frameworks to anchor the representation space from two distinct perspectives. In Adapt Without Forgetting: Distill Proximity from Dual Teachers in Vision-Language Models, Mengyu Zheng et al. employ two teachers: the original pre-trained model (which holds zero-shot general knowledge) and the model from the immediately preceding task.
┌──────────────────────────────┐
│ Original Pre-trained Teacher │ (Preserves General Robustness)
└──────────────┬───────────────┘
│
▼ Proximity Distillation
┌───────────────┐ ┌──────────────┐
│ Current Input │────────>│ Student Model│
└───────────────┘ └──────────────┘
▲ Proximity Distillation
│
┌──────────────┴───────────────┐
│ Immediate Past Teacher │ (Preserves Recent Task Skills)
└──────────────────────────────┘
By distilling the relative proximity of samples from both teachers, the student model learns how to accommodate new data patterns while maintaining the structural relationships of its original feature space. This strategy is also applied in few-shot scenarios via Few-Shot Class-Incremental Learning via Class-Aware Bilateral Distillation, where one teacher preserves base knowledge and another guides the integration of novel concepts, preventing the model from overfitting to small, recent datasets.
The Scaling Limits of Feature Learning
A natural question arises: does simply making a model larger solve the stability-plasticity dilemma? In The Importance of Being Lazy: Scaling Limits of Continual Learning, Jacopo Graldi investigates how model scale interacts with forgetting.
The paper demonstrates that wider networks only resist forgetting if the scaling pushes the network into a "lazy" training regime where internal features remain relatively static. If a model undergoes "rich" feature learning (where representations change significantly to optimize for tasks), scaling up the width does not inherently mitigate catastrophic forgetting. Instead, there exists a precise, scale-invariant threshold of feature learning that maximizes the trade-off between stability and plasticity—a balance that transfers predictably from small models to massive ones.