Continual Learning Without Data Rehearsal

511 concepts379 papers1 proposal

When a neural network learns tasks sequentially, it suffers from catastrophic forgetting. As the model optimizes its weights for a new task, it overwrites the representations required for previous ones. The most straightforward remedy is data rehearsal—maintaining a replay buffer of past data samples (exemplars) to interleave with new data during training.

However, in real-world applications, storing historical data is frequently impossible. Strict privacy regulations (such as GDPR or HIPAA) legally prohibit the long-term retention of sensitive user data, biometric information, or medical records. Additionally, hardware constraints on edge devices limit physical storage capacity.

These constraints have driven the development of exemplar-free continual learning (also known as data-free incremental learning). The core objective is to learn a continuous stream of new tasks or classes while completely discarding past data, ensuring total data privacy and minimal storage overhead.


Why Exemplar-Free Learning Fails by Default

To understand why exemplar-free learning is difficult, consider what happens to a model's latent space during sequential training. Without historical data to anchor the optimization process, two destructive phenomena occur:

  1. Representation Drift: As the network updates its weights to minimize loss on a new task, it warps the underlying feature space. Features that once cleanly separated old classes collapse or shift, rendering the old decision boundaries obsolete.
  2. Classifier Bias: The final classification layer (the head) experiences severe data imbalance. Because it only receives gradients from the new classes, its weights and biases grow disproportionately large for those new categories, effectively blinding the model to historical classes.

To overcome these challenges without a replay buffer, researchers have developed four primary strategies: geometric constraints, analytic updates, functional regularization, and parameter isolation.


Geometric Constraints and Orthogonal Subspaces

One intuitive way to prevent forgetting without storing data is to constrain the optimization path geometrically. Instead of allowing gradients to change weights arbitrarily, we can restrict updates to directions that do not disrupt past tasks.

Imagine the network's parameter space as a high-dimensional landscape. A task utilizes specific directions (subspaces) within this landscape to extract features. If we can identify these subspaces, we can force all future weight updates to occur only in directions orthogonal to them.

Standard Update:      New Gradient ───> Alters Past Representations

Orthogonal Update:    New Gradient ───> Projected onto Orthogonal Complement ───> Preserves Past Subspaces

Saha et al. formalized this mechanism via Gradient Projection Memory (GPM) in Gradient Projection Memory for Continual Learning. By performing Singular Value Decomposition (SVD) on network activations at the end of a task, the authors identify the essential mathematical subspace responsible for that task's performance. When a new task arrives, the optimization gradients are projected onto the orthogonal complement of this stored subspace, ensuring that the model's behavior on old tasks remains unchanged.

This geometric approach scales effectively to distributed environments where privacy constraints are even tighter. For instance, Moussadek et al. leverage this concept in DOLFIN: Balancing Stability and Plasticity in Federated Continual Learning. By combining orthogonal low-rank adapters (LoRA) with a gradient projection strategy, the authors allow decentralized clients to learn sequentially without exchanging or storing raw data or class prototypes.


Analytic Learning and Recursive Matrix Updates

Rather than relying on iterative gradient descent—which inherently risks overwriting past weights—some approaches bypass backpropagation entirely for the final layers using analytic learning.

The intuition here is rooted in linear algebra: if a classification problem can be framed as a recursive least-squares or linear ridge regression problem, we do not need the raw data to compute the optimal weights. We only need the data's statistical footprint, such as feature covariance matrices. These matrices are compact, fixed in size, and contain no identifiable raw data, making them perfectly privacy-preserving.

Zhuang et al. demonstrate this mathematical exactness in ACIL: Analytic Class-Incremental Learning with Absolute Memorization and Privacy Protection. They prove that recursive linear updates can achieve "absolute memorization"—mathematically matching the performance of training on all historical data simultaneously—without storing a single raw image.

This framework has been expanded to handle more complex, real-world data structures:


Functional Regularization and Data-Free Replay

If we cannot store real data, can we use the model itself to preserve or recreate it? This is the core intuition behind functional regularization and generative data-free replay.

Refining Knowledge Distillation

Knowledge distillation is widely used to force a new model's outputs to match a frozen copy of the old model. However, standard distillation can sometimes conflict with learning new boundaries. For example, in metric learning, forcing the model to keep positive pairs close together can restrict its ability to separate new classes.

To resolve this, Wang et al. proposed Distillation without Positive Pairs (DwoPP) in Positive Pair Distillation Considered Harmful: Continual Meta Metric Learning for Lifelong Object Re-Identification. The authors demonstrate that removing positive-pair constraints prevents distillation from fighting against the metric learning objectives, allowing the embedding space to remain stable without storing raw images.

Generative Synthesis

Alternatively, we can train a generator alongside our model to synthesize pseudo-exemplars. Instead of saving real user data, the system learns the underlying distribution and "dreams up" synthetic data to act as a proxy during rehearsal.

[Old Model] ───> Guides ───> [Generator] ───> Synthesizes "Uncertain" Samples ───> [New Model Training]

Ye et al. developed this into a formal framework called the Self-distilled Knowledge Delegator (SKD) in Self-distilled Knowledge Delegator for Exemplar-free Class Incremental Learning, using synthetic samples to delegate knowledge from old network versions to new ones. To make this synthesis more efficient, Liu et al. developed a method detailed in Few-Shot Class-Incremental Learning via Entropy-Regularized Data-Free Replay that forces the generator to synthesize "uncertain" samples located near the decision boundaries of old classes. This explicitly reinforces the most vulnerable regions of the latent space without exposing actual historical data.


Parameter Isolation and Pre-trained Foundation Models

The rise of powerful pre-trained models (PTMs) like Vision Transformers (ViTs) and CLIP has introduced a completely different strategy: if the base features are already highly generalizable, we do not need to change them at all. Instead, we can isolate parameters.

By freezing the core weights of a large pre-trained model, we completely eliminate representation drift. To learn new tasks, we append small, task-specific modules (like prompts or adapters) that steer the frozen representations.

Zhou et al. demonstrate the power of this paradigm in MOS: Model Surgery for Pre-Trained Model-Based Class-Incremental Learning. They show that merging task-specific adapters during training, combined with a training-free self-correction step at inference, can eliminate forgetting entirely. To optimize this process for speed, Gao et al. introduced Beyond Prompt Learning: Continual Adapter for Efficient Rehearsal-Free Continual Learning, which bypasses slow prompt-matching mechanisms in favor of parameter-expandable adapter layers.

However, if there is a large "semantic gap" between the model's original pre-training and the new tasks, static prompts fail. To address this, Tang et al. proposed an Adaptive Prompt Generator in When Prompt-based Incremental Learning Does Not Meet Strong Pretraining to dynamically unify prompt retrieval and learning, allowing exemplar-free learning to succeed even when the underlying pre-training is weak.

Finally, this isolation strategy extends beyond classification to open-world discovery. In the context of intent recognition, Song demonstrates in Continual Generalized Intent Discovery: Marching Towards Dynamic and Open-world Intent Recognition that class prototypes and feature distillation can be combined to discover entirely new user intents over time, paving the way for zero-replay discovery systems that maximize user privacy.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 390 papers
  2. Version 1 · 2026-10-04 · 340 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_exemplar_free_incremental_learning_8fc9340e,
  title = {exemplar-free incremental learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/exemplar-free-incremental-learning/rdr_8fc9340ee165492297741fc5e59a9a05},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 2, frozen text):

@misc{lacuna_exemplar_free_incremental_learning_8fc9340e_v2,
  title = {exemplar-free incremental learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/exemplar-free-incremental-learning/rdr_8fc9340ee165492297741fc5e59a9a05/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake