Orthogonal Subspaces and Parameter Isolation for Continual Learning

558 concepts496 papers2 proposals

When a neural network learns a sequence of tasks, it faces a fundamental conflict known as the stability-plasticity dilemma. To learn a new task, the network must be plastic—its weights must change to minimize the new loss. However, to retain past knowledge, it must be stable—its weights must remain unchanged to preserve the loss minima of previous tasks.

In standard deep learning, gradient descent is greedy. When optimizing for a new task, gradients backpropagate through the entire network, overwriting the weight configurations carefully carved out by prior data. This results in catastrophic forgetting, where the model's performance on earlier tasks abruptly collapses.

To resolve this dilemma, recent research has shifted away from simply penalizing weight changes. Instead, modern approaches focus on geometric and structural strategies: physically isolating parameters for different tasks, projecting weight updates into orthogonal subspaces where they cannot interfere with past knowledge, or actively managing the geometry of the latent space.


Parameter Isolation: Carving Out Dedicated Capacities

The most direct way to prevent a new task from overwriting an old one is to ensure they do not share the same trainable parameters. By freezing the core weights of a pre-trained model and allocating small, task-specific parameters (often called adapters), we can guarantee perfect stability for historical tasks while maintaining plasticity for new ones.

However, adding a new set of parameters for every incoming task quickly leads to a memory explosion. To make parameter isolation viable for long sequences of tasks, recent methods focus on extreme compactness and intelligent parameter reuse.

For instance, Tasnim et al. introduce RECAST, a framework that decomposes network weights into frozen shared templates and a tiny set of learnable coefficients. By adapting only these coefficients, the model can learn sequential tasks with fewer than 50 trainable parameters per task, drastically reducing the memory footprint on edge devices.

Instead of blindly allocating new parameters for every task, we can also dynamically decide when an old adapter is "good enough" to be reused. Julian et al. propose CABLE, a reinforcement learning framework that computes the gradient similarity between new incoming data and past tasks. If the similarity is high, the system reuses an existing adapter, promoting forward knowledge transfer; if the data represents an entirely novel distribution, it initializes a new adapter.

Task 1 Data ---> [ Shared Frozen Backbone ] ---> [ Adapter A (Tuned) ]
                                                       |
Task 2 Data ---> [ Shared Frozen Backbone ] ---> [ Adapter B (New)   ] (Isolated)

Similarly, Yoon et al. utilize a hybrid approach in XtarNet, which extracts task-adaptive representations by dynamically mixing features from a fixed, pre-trained backbone with meta-learned novel features. This ensures that base categories remain perfectly stable while new categories benefit from targeted plasticity.


Orthogonal Subspaces: Learning Without Interference

What if we must update the shared weights of a network? Is it possible to modify a weight matrix to learn Task B without altering its behavior on Task A?

Mathematically, the answer lies in the null space of past tasks. If the weight updates $\Delta W$ for a new task are strictly orthogonal to the input features $X_{\text{old}}$ utilized by previous tasks, then:

$$X_{\text{old}}(W + \Delta W) = X_{\text{old}}W + 0$$

The model's outputs for historical data remain completely unchanged, achieving zero interference.

Li et al. leverage this geometric property in InfLoRA (Interference-Free Low-Rank Adaptation). InfLoRA mathematically projects parameter updates into a low-rank subspace designed to be orthogonal to the feature spaces of old tasks but aligned with the current task. This eliminates catastrophic forgetting without requiring the storage of raw historical data or increasing inference latency.

Old Task Gradient Space  <---|--->  New Task Gradient Space
                             |
                      (Orthogonal)

To provide rigorous theoretical guarantees for this behavior, Peng et al. introduce LoRanPAC. By combining large pre-trained models with an incremental truncated Singular Value Decomposition (SVD) approach, the authors construct a sequential learning system with provable bounds against numerical instability and catastrophic forgetting, bridging the gap between empirical performance and theoretical stability.

Rather than constraining the weight matrices directly, we can also enforce orthogonality and stability within the network's functional outputs. Gomez-Villa et al. propose Projected Functional Regularization (PFR), which uses a temporal projection network to map newly learned feature representations back into the historical feature space. This allows the model to adapt its internal representations to new data streams while ensuring that the global geometry of the embedding space remains stable.


Latent Space Geometry: Prototypes and Repulsion

In class-incremental learning—where a model must learn to classify entirely new categories over time—catastrophic forgetting often manifests as task confusion. Because the model is only trained on new classes during a given session, the decision boundaries drift, causing the model to over-bias its predictions toward the most recently seen categories.

When storing raw data from past tasks is prohibited due to privacy or memory constraints, managing the geometry of the latent space via class prototypes becomes essential. A prototype is a compact representation (typically the mean vector) of a class in the embedding space.

Nori et al. address task confusion by introducing Autoencoder-Based Hybrid Replay (AHR). Instead of saving raw images, they compress past data into a latent space. To prevent new classes from encroaching on old class territories, they implement a physics-inspired Repulsive Force Algorithm. This algorithm actively pushes new class clusters away from historical prototypes, maintaining clean, well-separated decision boundaries.

[ Old Class Cluster ]  <--- (Repulsive Force) --->  [ New Class Cluster ]

Taking this concept to its logical limit, Kong et al. develop YoooP (You Only Optimize One Prototype), a non-exemplar incremental learning method. By optimizing exactly one highly representative prototype per class and pairing it with a rotation-based data synthesis technique, the authors preserve the structural integrity of the historical feature space without storing a single original sample.

Even under messy, real-world conditions where data labels are ambiguous, prototypes can act as an anchor for stability. Wang et al. introduce PGDR, a framework designed for incremental learning under partial or ambiguous supervision. PGDR uses stable class prototypes to separate old and new data distributions, refining ambiguous labels via momentum-based pseudo-labeling while using a diverse memory buffer to keep the latent space from collapsing.


Conclusion

The stability-plasticity dilemma is not an insurmountable flaw of gradient descent, but a geometric constraint. As demonstrated by recent literature, we can bypass catastrophic forgetting by changing how and where updates occur in a network. Whether by isolating parameters via compact templates (Tasnim et al.), projecting updates onto orthogonal null spaces (Li et al.), or enforcing geometric repulsion between latent clusters (Nori et al.), modern continual learning succeeds by ensuring that new knowledge expands the model's capabilities without intersecting—or overwriting—the past.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_stability_plasticity_dilemma_catastrophi_4f2bc08e,
  title = {stability-plasticity dilemma (catastrophic forgetting)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/stability-plasticity-dilemma-catastrophic-forgetting/rdr_4f2bc08e2c8d4c37b5f0f9af7ea8a954},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_stability_plasticity_dilemma_catastrophi_4f2bc08e_v1,
  title = {stability-plasticity dilemma (catastrophic forgetting)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/stability-plasticity-dilemma-catastrophic-forgetting/rdr_4f2bc08e2c8d4c37b5f0f9af7ea8a954/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake