When a deep learning model learns new tasks sequentially, it faces the stability-plasticity dilemma. To retain old knowledge (stability), models often employ Knowledge Distillation (KD), forcing the current "student" model to mimic a historical "teacher" model. To learn new concepts (plasticity), the model must update its weights based on new data.
Standard knowledge distillation applies a uniform penalty across all outputs or features. However, this global constraint creates a severe failure mode, particularly in dense prediction tasks like object detection or semantic segmentation. When new classes appear in regions that the old model considered "background," global distillation forces the student to maintain the teacher's outdated background predictions. This over-constrains the network, crushing its plasticity and causing representational conflict.
To solve this, a powerful paradigm has emerged: Selective Feature Distillation. Instead of a blanket regularization signal, these techniques surgically apply distillation constraints across spatial, channel-wise, or data-driven boundaries. By protecting only what is truly critical to past knowledge, they free up the rest of the network to adapt.
Spatial and Regional Masking: Distilling Where Old Knowledge Lives
The most intuitive way to decouple stability and plasticity is to look at where features occur. In an image or video, old categories and new categories rarely occupy the exact same pixels. If we can isolate the spatial regions belonging to historical classes, we can restrict distillation exclusively to those zones.
Global Distillation (Rigid) Selective Masked Distillation (Plastic)
[ Teacher ] [ Student ] [ Teacher ] [ Student ]
│ │ │ │
▼ ▼ ▼ ▼
┌──────────────────────┐ ┌───────────────────────────────┐
│ Distill Entire Image │ │ Distill ONLY Past-Class Zones │
└──────────────────────┘ └───────────────────────────────┘
(Suppresses new foreground) (Allows new classes to emerge)
In class-incremental semantic segmentation, Jinpeng Chen et al. demonstrate in Saving 100x Storage: Prototype Replay for Reconstructing Training Sample Distribution in Class-Incremental Semantic Segmentation that applying constraints solely to spatial regions predicted to contain old classes maintains feature stability without making the network overly rigid. Similarly, Chao Shang et al. introduce a spatially masked distillation loss in Incrementer: Transformer for Class-Incremental Semantic Segmentation with Knowledge Distillation Focusing on Old Class to prevent the model from incorrectly treating new objects as background.
This principle is equally vital for object detection. In Incremental-DETR: Incremental Few-Shot Object Detection via Self-Supervised Learning, Na Dong et al. utilize a binary mask derived from the bounding boxes of novel objects. This mask explicitly disables distillation in foreground regions where new objects reside, ensuring that the teacher's base-class knowledge does not conflict with the student's acquisition of new object features.
For modern Vision Transformers (ViTs), this spatial selectivity translates to patch-level modulation. Jiashuo Li et al. show in Dynamic Integration of Task-Specific Adapters for Class Incremental Learning that stabilizing individual image patches that do not contribute to new tasks mitigates catastrophic forgetting in non-exemplar environments. To refine this, Jiang-Tian Zhai et al. propose in Fine-Grained Knowledge Selection and Restoration for Non-exemplar Class Incremental Learning to weight patch distillation based on a patch's relevance to the classification token, maintaining stability on task-agnostic background features while preserving plasticity for foreground objects.
Even when moving from 2D images to evolving 3D scenes, the mechanism holds. In CL-NeRF: Continual Learning of Neural Radiance Fields for Evolving Scene Representation, Xiuzhe Wu et al. implement mask-conditioned self-distillation. By selectively disabling the distillation signal in areas where the scene has physically changed, the model preserves legacy knowledge in static regions without interfering with new observations.
Channel-wise and Feature Importance: Protecting Critical Representations
Not all latent features or convolutional channels contribute equally to a given piece of knowledge. While some channels encode highly specific discriminative features for past tasks, others are redundant or underutilized. Selective distillation can therefore operate along the channel dimension, identifying and protecting only the critical pathways.
To identify these pathways, Jaeyoo Park et al. introduce "time-channel importance maps" in Class-Incremental Learning for Action Recognition in Videos. By measuring the sensitivity of the classification loss to specific feature changes, the authors prioritize the most discriminative feature channels and time steps, ensuring essential video knowledge is not overwritten.
Once feature importance is established, it can be used to dynamically scale the distillation intensity:
- Attention-Guided Scaling: Guanglei Yang et al. apply channel-wise and spatial attention weights directly to the loss function in Continual Attentive Fusion for Incremental Learning in Semantic Segmentation, ensuring that the most significant features for old classes are prioritized.
- Decoupled Optimization: In CoMBO: Conflict Mitigation via Branched Optimization for Class Incremental Segmentation, Kai Fang et al. calculate the significance of individual features relative to historical classes. Critical features face strict preservation constraints, while less relevant features are granted the flexibility to adapt to new information.
Resolving Gradient Conflicts
What happens when the gradient to learn a new task directly opposes the gradient required to preserve an old one? Jisu Han et al. tackle this in SRIL: Selective Regularization for Class-Incremental Learning via gradient-aligned selective distillation. By calculating the cosine similarity between the gradients of the classification loss and the knowledge distillation loss on a per-channel basis, the authors identify conflicts. When the gradients point in opposite directions, the distillation constraint is masked out, allowing the model to learn unique features for new classes without being bottlenecked by historical representations.
Data-Driven and Subspace Selection: Purifying the Distillation Signal
A final dimension of selectivity focuses on when or on what data to apply distillation. Distilling over the entire incoming data distribution can inadvertently force the model into suboptimal representations for novel inputs.
For instance, in speech processing, Umberto Cappellazzo et al. demonstrate in An Investigation of the Combination of Rehearsal and Knowledge Distillation in Continual Learning for Spoken Language Understanding that restricting feature-level distillation exclusively to rehearsal data (and omitting it on entirely new data) prevents the model from being overly constrained, allowing it to form distinct feature clusters for new tasks.
Selectivity can also be governed by the model's own confidence or structural properties:
- Semantic Confidence: In weakly supervised settings, Yu-Hsing Hsieh et al. show in Class-incremental Continual Learning for Instance Segmentation with Image-level Weak Supervision that distilling knowledge only in regions where the teacher model exhibits high confidence prevents outdated or noisy background predictions from suppressing new classes.
- Subspace Purification: In Towards Realistic Lifelong Re-identification: Identity Recurrence with Changing Clothes, Wuxuan Shi et al. restrict knowledge distillation to a purified identity subspace rather than the full feature set. This allows the model to adapt to transient variations (like changing clothes) while firmly anchoring core identity markers.
Why It Works: A Unified Intuition
Selective feature distillation works because it treats a neural network's capacity as a dynamically allocatable resource. Global distillation treats the network as a single monolithic block that must be frozen in time. By breaking down the distillation objective along spatial, channel, and data boundaries, these methods recognize that knowledge is localized.
By applying strict regularization only to the specific coordinates—whether they are image pixels, transformer patches, or latent channels—that underpin historical performance, the stability-plasticity dilemma is resolved. The model retains its past with high fidelity, while the unconstrained regions of the parameter space remain completely plastic, ready to absorb new information.
Key Foundational Work
Always Be Dreaming: A New Approach for Data-Free Class-Incremental Learning (2022) — contributed 2 concept elements to this direction.
Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidation (2022) — contributed 2 concept elements to this direction.
Class-Incremental Learning for Action Recognition in Videos (2021) — contributed 1 concept elements to this direction.