When a neural network learns a new task, it typically undergoes "catastrophic forgetting," where the weights shift so drastically to accommodate new data that the model loses its ability to perform previous tasks. While early solutions focused on replaying old data or freezing specific weights, a more robust and flexible direction has emerged: Feature Representation Stabilization.
Instead of trying to keep the weights themselves static, this approach focuses on keeping the model's internal logic—its latent feature space—consistent over time. By ensuring that the way a model "sees" and embeds information does not drift, researchers allow the network to remain plastic enough to learn new concepts while keeping its historical knowledge intact.
The Stability-Plasticity Dilemma as Representation Drift
The fundamental challenge in continual learning is the stability-plasticity dilemma. As Saihui Hou and Chen Change Loy identify in Learning a Unified Classifier Incrementally via Rebalancing, the primary culprit behind model bias is the imbalance between new and old data, which causes the feature extractor to drift.
When the feature extractor drifts, the "coordinates" of old classes in the latent space change. An old classifier head, which was trained to look for a specific pattern at a specific location in that space, suddenly finds itself looking at a vacuum or a different class entirely. Stabilization techniques aim to "anchor" these coordinates.
Anchoring the Latent Space via Distillation
The core mechanism for stabilization is Feature-based Knowledge Distillation. In this setup, the model from the previous training phase acts as a "teacher" for the current "student" model.
Unlike standard distillation, which often only matches the final output probabilities (logits), this research direction prioritizes matching the intermediate layers. As Jianming Zhang and colleagues demonstrate in M2KD, distilling knowledge from intermediate layers and auxiliary classifiers helps maintain stable feature representations across the entire network backbone. This ensures that the fundamental visual or semantic patterns learned early on remain available for future tasks.
This anchoring effect is visible across diverse domains:
- Robotics: NBAgent uses representation distillation on 3D voxel features to prevent the degradation of spatial knowledge as a robot learns new manipulation skills.
- Medical Imaging: MedSeqFT employs distillation to ensure that 3D segmentation models maintain stable internal representations when moving between different clinical tasks.
- Intent Detection: Guirong Bai uses multilevel distillation in Contrast Replay Networks to prevent the model's internal logic from being overwritten by technical medical terminology.
Geometric Constraints and Normalization
A key intuition in this field is that the orientation of features often matters more than their magnitude. Learning a Unified Classifier Incrementally via Rebalancing introduced cosine normalization to constrain the orientation of internal feature representations. By preserving the geometric structure of the feature space, the model ensures that existing classifier weights remain valid for previously learned classes.
Similarly, Abdelrahman Mohamed proposes D3Former, which uses a dual distillation strategy to preserve both feature representations and spatial attention maps. This ensures that the model not only remembers what it saw but also where it was looking, preventing the "attention drift" that often precedes catastrophic forgetting in Transformers.
Contrastive and Multi-level Refinements
Recent work has moved toward more sophisticated alignment signals. Standard distillation can sometimes be too rigid, preventing the model from learning new things effectively. To solve this, researchers like Barbara Caputo have introduced contrastive regularization in A Contrastive Distillation Approach for Incremental Semantic Segmentation.
Instead of forcing the student to be an exact copy of the teacher, contrastive distillation forces the student to maintain the relational structure of the data. If two images were close together in the teacher's feature space, they should remain close in the student's space, even if the entire space has shifted slightly to accommodate new classes. This "relational stability" provides a more flexible signal that resists the drift typically caused by new data distributions.
Why it Works: The Preservation of Causal Structure
The ultimate goal of feature stabilization is to capture and retain domain-invariant representations. As argued in Continual Learning of Domain-Invariant Representations, distillation acts as a guardian of the "invariance signatures" recorded when data was first processed.
By penalizing deviations from these signatures, the model is forced to rely on universal features (like shapes or semantic structures) rather than shortcuts (like background colors or specific dataset biases). This makes the model not just better at remembering the past, but more robust when facing an unpredictable, "open-world" future.