Artificial neural networks are notoriously "forgetful." When trained on a sequence of tasks, they suffer from catastrophic forgetting: the weights optimized for Task A are overwritten to satisfy Task B. While the most effective defense has traditionally been "experience replay"—storing a buffer of old data to periodically remind the model of what it once knew—this approach is increasingly untenable.
As noted by Gobinda Saha and Kaushik Roy, storing raw data creates massive storage overhead and, more critically, violates the strict privacy regulations (like GDPR) common in medical or personal finance domains. The research community is therefore pivoting toward rehearsal-free continual learning: techniques that allow a model to grow its knowledge base without ever looking back at original samples.
Geometric Isolation via Orthogonal Projections
One of the most intuitive ways to prevent forgetting is to ensure that new learning happens in a "space" that the old tasks aren't using. Neural networks are massive and highly overparameterized; for any given task, there are many directions in the weight space that the model doesn't actually need.
In Gradient Projection Memory (GPM), Gobinda Saha and colleagues use Singular Value Decomposition (SVD) on network activations to identify the essential subspaces (the "important directions") for a task. When a new task arrives, the model's gradients are projected so they are orthogonal to these past subspaces.
Think of it like a high-dimensional room: if Task A occupies the floor, Task B is constrained to only move along the walls. Because the walls are orthogonal to the floor, Task B’s updates have zero mathematical impact on the logic Task A uses to make decisions. This concept is further refined in Hierarchical Visual Prompt Learning (HVPL) by Jiahua Dong, which applies these orthogonal constraints to the prompts of vision transformers.
Structural Isolation and Parameter Masking
If geometric isolation is about moving in different directions, structural isolation is about using different "wires." Researchers have found that not every parameter is equally important for every input.
Wenyu Du and the team behind MIGU (Unlocking Continual Learning Abilities in Language Models) discovered that by only updating parameters that produce the largest output magnitudes for a given task, they can create a natural, task-specific mask. This isolates knowledge within specific circuits of the model. Similarly, Yabin Wang and Zhiheng Ma proposed Energy Self-Normalization (ESN), which trains isolated classifiers on a frozen backbone. By using an energy-based metric to aggregate these "experts," the model can learn indefinitely without the interference that causes forgetting.
Synthetic Rehearsal and Generative Reminiscence
When raw data cannot be stored, some researchers suggest the model should "dream" of its past instead. This is known as generative or pseudo-rehearsal.
In the RECALL framework, Andrea Maracani uses Generative Adversarial Networks (GANs) to "hallucinate" samples of old classes. Instead of storing a photo of a cat from Task 1, the model learns the distribution of cats and generates a new, synthetic cat image to train alongside Task 2's data. Chaitanya Ahuja takes this further in C-DiffGAN for gesture generation, creating "reminiscences" of previous speakers to align new learning with old styles without retaining sensitive personal recordings.
Mathematical Guarantees via Weight Intervals
While many methods are heuristic, some researchers seek absolute certainty. Maciej Wolczyk and Bartosz Wójcik introduced InterContiNet, which treats continual learning as a constrained optimization problem.
Instead of finding a single point in weight space, they find a "hyperrectangle"—an interval of valid values for each weight—that satisfies the current task. As the model learns new tasks, it is mathematically restricted to stay within the intersection of all previous hyperrectangles. This provides a formal guarantee: as long as a solution exists within those bounds, the model's performance on Task 1 will never degrade, effectively "locking in" skills as they are acquired.
Representation Stability
Finally, some methods focus on the "drift" of internal representations. Lucas Caccia and Eugene Belilovsky demonstrated with ER-ACE that forgetting is often caused by new data "pushing" old class representations out of their clusters. By simply masking out the output logits of old classes during new training, they prevent this drift.
Similarly, Jary Pomponi and Vincenzo Lomonaco proposed Embedding Regularization, which penalizes the model if its internal "understanding" of an input changes too much between tasks. By keeping the latent space stable, the model can layer new concepts on top of old ones without the foundation crumbling.