In modern machine learning, we often treat a model’s internal representation of data as a fixed map. We assume that once a model learns that "dogs" belong in a specific region of its feature space, that region remains the definitive home for dogs. However, in dynamic scenarios like continual learning or domain adaptation, this map is constantly being redrawn.
As a neural network learns new tasks or adapts to new environments, its weights change. This causes the underlying feature space to warp, shift, and rotate—a phenomenon known as feature drift or semantic drift. When the map changes but our "landmarks" (class prototypes or classifier weights) stay in their old positions, the model suffers from catastrophic forgetting.
Researchers have converged on a shared solution: instead of keeping landmarks static, we must dynamically compensate for drift to keep prototypes synchronized with the evolving feature space.
The Moving Target Problem
The core challenge, as identified by Bo-Cheng Qiu, Chia-Wen Lin, and Chih-Chung Hsu in CNN-Based Joint Clustering and Representation Learning with Feature Drift Compensation for Large-Scale Image Data, is that joint learning is inherently unstable. When you update a network to better cluster data, the very coordinates of that data change.
In incremental learning, this problem is even more acute. When a model trained on Task A is fine-tuned on Task B, the features for Task A classes "drift" away from their original centroids. Piotr Koniusz and Can Peng argue in Multivariate Prototype Representation for Domain-Generalized Incremental Learning that static representations of old classes become obsolete almost immediately as the backbone evolves.
Estimating Drift Without Old Data
The most elegant solutions to this problem are "exemplar-free"—they don't require storing old images, which is often impossible due to privacy or storage constraints. But if you don't have the old data, how do you know how much the feature space has shifted?
The field has developed several clever "proxy" mechanisms:
- Current Data as a Compass: Lu Yu and Bartłomiej Twardowski introduced Semantic Drift Compensation (SDC) in Semantic Drift Compensation for Class-Incremental Learning. They use the movement of new task data in the embedding space to estimate the displacement of old class prototypes.
- Adversarial Proxies: In Resurrecting Old Classes with New Data for Exemplar-Free Continual Learning, Dipam Goswami and colleagues use adversarial attacks to turn new images into "pseudo-exemplars" of old classes. By seeing how these proxies move between model versions, they calculate a geometric shift to apply to old prototypes.
- Analogical Prompts: Zhiheng Ma and Xiaopeng Hong use "analogical prompts" in Remind of the Past: Incremental Learning with Analogical Prompts to imagine how new data would look to the old model, creating a bridge between the two feature spaces.
Geometric Realignment Strategies
Once the drift is estimated, the next question is how to apply the correction. Simple vector addition is often insufficient because the feature space doesn't just shift; it deforms.
In Prototypical quadruplet for few-shot class incremental learning, Biplab Banerjee and Subhasis Chaudhuri propose using statistical whitening and re-coloring. By adjusting the mean and variance of class distributions, they ensure historical anchors remain aligned with the current state of the feature extractor.
Other researchers use more complex transformations. Donghyeon Baek and Bumsub Ham utilize category-specific rotation matrices in ALIFE: Adaptive Logit Regularizer and Feature Replay for Incremental Semantic Segmentation to preserve the internal cluster structures of old categories as weights shift. Similarly, Hiroshi Kera uses affine corrections in Adapter Merging with Centroid Prototype Mapping for Scalable Class-Incremental Learning to realign representations after merging specialized model components.
Real-time and Test-time Adaptation
The most recent frontier in this research direction is moving drift compensation from the training phase to the inference phase.
Test-Time Classifier Adjustment (T3A), proposed by Yusuke Iwasawa and Yutaka Matsuo in Test-Time Classifier Adjustment Module for Model-Agnostic Domain Generalization, demonstrates that we can adjust classifier weights on the fly. By calculating "pseudo-prototypes" from the incoming stream of unlabeled target data, the model can realign its decision boundaries to a new environment without any backpropagation.
This online philosophy is echoed in the work of Samantha Guerriero and Thomas Mensink. Their DeepNCM framework (DeepNCM: Deep Nearest Class Mean Classifiers) uses exponential moving averages to track drifting class centers continuously, ensuring that the "landmarks" are never more than a step behind the "map."
Why It Works: The Intuition of Stability
The success of these methods across image recognition, graph learning, and natural language processing suggests a fundamental principle: Classification is relative, not absolute.
A model doesn't need a static coordinate for "cat"; it needs a coordinate that is consistently positioned relative to the features the encoder is currently producing. By treating prototypes as dynamic entities that can be transported (Pascal Mettes, Universal Prototype Transport for Zero-Shot Action Recognition and Localization) or predicted (Daniel Marczak, Generalized Continual Category Discovery), we decouple the stability of the classifier from the plasticity of the feature extractor. This allows models to learn and adapt indefinitely without losing their grip on the past.