Imagine a neural network trained to recognize cats and dogs. It maps images into a latent feature space where cats cluster on the left and dogs cluster on the right. To save memory, we compute the average vector—the prototype—for each class and discard the raw images.
Now, the network must learn a new task: recognizing birds. As the network updates its weights to extract bird features, the underlying geometry of the latent space warps. The coordinates that once cleanly isolated cats and dogs have shifted. The stored prototypes are now "stale," pointing to outdated locations in a reconfigured space. This phenomenon is known as feature drift or representation drift, and it is a primary driver of catastrophic forgetting in continual learning.
Instead of freezing the network (which destroys plasticity) or storing massive amounts of old data (which violates memory and privacy constraints), a powerful family of techniques focuses on feature drift compensation. The core intuition is simple: if we can estimate how the feature space shifted, we can mathematically update the old prototypes to match the new geometry.
Estimating the Shift Without Old Data
The fundamental challenge of exemplar-free continual learning is tracking how the feature space changes when you no longer have access to the old data. How do you measure a transformation if you cannot observe the points being transformed?
The key insight is to use the current task's data as a proxy. By passing new data through both the old model (frozen from the previous task) and the current evolving model, we can observe the representational shift in real time.
For instance, Ahmet Iscen et al. in Memory-Efficient Incremental Learning Through Feature Adaptation demonstrate that the mapping between successive feature spaces can be generalized by observing how current task representations shift between model versions. A dedicated transformation module can then translate stored feature descriptors into the updated space.
Similarly, Piotr Koniusz et al. extend this to domain-generalized settings in Multivariate Prototype Representation for Domain-Generalized Incremental Learning, estimating feature space drift by comparing outputs of the current and previous model versions on new data to update the parameters of stored multivariate normal distributions.
[Old Model] ---> (Current Data) ---> Old Feature Space \
+---> Calculate Shift Matrix
[New Model] ---> (Current Data) ---> New Feature Space /
When current data is structurally distinct from historical data, researchers turn to synthetic or adversarial proxies. Dipam Goswami et al. propose Resurrecting Old Classes with New Data for Exemplar-Free Continual Learning, which uses targeted adversarial attacks to transform new images into "pseudo-exemplars" of old classes. By comparing the embeddings of these pseudo-exemplars across the old and new model versions, they calculate the exact geometric shift and apply it directly to relocate the old prototypes.
Mathematical Formulations of Drift
Drift is rarely uniform across the entire feature space; it can involve global translations, rotations, or complex category-specific deformations. To correct for this, methods model the drift using varying degrees of geometric complexity:
- Global and Category-Specific Projections: Run He et al. in Semantic Shift Estimation via Dual-Projection and Classifier Reconstruction for Exemplar-Free Class-Incremental Learning show that semantic shift can be accurately estimated by combining a task-wide linear transformation with category-specific information projections. By utilizing singular value decomposition (SVD) on class covariances, they preserve fine-grained details while adapting old knowledge to the new feature space.
- Rotation Matrices: In the context of semantic segmentation, Donghyeon Baek et al. introduce ALIFE: Adaptive Logit Regularizer and Feature Replay for Incremental Semantic Segmentation, which employs category-specific rotation matrices to align stale latent features with the current state of an evolving feature extractor, preserving cluster structures with minimal memory overhead.
- Statistical Alignment: Rather than treating prototypes as static vectors, Biplab Banerjee et al. in Prototypical quadruplet for few-shot class incremental learning dynamically adjust class centers using statistical whitening and re-coloring transformations based on mean and variance statistics, ensuring historical class anchors remain accurate relative to the evolving network.
Dynamic Prototype Realignment
Once the drift is estimated, the system must update the stored representations. A common strategy is to dynamically interpolate or shift the prototypes to maintain topological relationships.
In Prototype Reminiscence and Augmented Asymmetric Knowledge Aggregation for Non-Exemplar Class-Incremental Learning, the authors argue against treating prototypes as static vectors. Instead, they evolve them by interpolating between stored historical averages and current feature representations, ensuring class boundaries remain accurate as the underlying space shifts.
For graph-based data, Peyman Moghadam et al. introduce a framework in Inductive Graph Few-shot Class Incremental Learning (GFSCIL) that utilizes iterative prototype calibration to compensate for feature drift when the model's backbone is fine-tuned for new graph topologies.
This concept also extends to generative models. Tao Jin et al. apply this to text-to-image synthesis in ConceptGuard: Continual Personalized Text-to-Image Generation with Forgetting and Confusion Mitigation. They implement dynamic embedding offsets that evolve alongside the model weights, ensuring that text tokens continue to point to the correct visual representations throughout long sequences of learning tasks.
Why It Works: Balancing Plasticity and Stability
The ultimate goal of feature drift compensation is to decouple the learning of new features from the preservation of old ones. Traditional methods like weight regularization penalize changes to the network parameters, which inherently limits the model's capacity to learn highly distinct new tasks (low plasticity).
As Run He et al. demonstrate, maintaining backbone plasticity while applying post-hoc mathematical corrections for feature drift provides a superior balance between stability and learning capacity. The network remains free to optimize its weights for the current task, while the projection and compensation mechanisms handle the retrospective alignment of historical knowledge. By shifting the burden of memory from parameter constraints to geometric transformations, feature drift compensation allows deep networks to continuously adapt without losing their past.
Key Foundational Work
CNN-Based Joint Clustering and Representation Learning with Feature Drift Compensation for Large-Scale Image Data (2017) — contributed 2 concept elements to this direction.