Introduction
Artificial intelligence systems often need to learn new categories over time from very few examples—a task known as Few-Shot Class-Incremental Learning (FSCIL). The primary hurdle in this domain is catastrophic forgetting: as a model learns new "novel" classes, it tends to overwrite the parameters necessary to recognize "base" classes it learned previously.
Existing solutions frequently employ Knowledge Distillation (KD) to align a "student" model's outputs with a "teacher" model (a previous iteration of itself). However, standard KD (Individual KD) often falters when data is scarce due to low-quality feature production. While Relational KD (RKD) attempts to address this by modeling inter-sample similarities, it frequently collapses complex structural information into a single dimension. This loss of directional data makes the model hypersensitive to outliers and prone to "structural collisions."
Displacement Knowledge Distillation (DKD)
To overcome the limitations of similarity-based distillation, this paper introduces Displacement Knowledge Distillation (DKD). Unlike RKD, which calculates a scalar similarity between samples, DKD calculates the displacement vector (the point-wise difference) between pairs of samples.
By preserving both the distance and the angular (directional) relationship in the original feature space, DKD ensures that no structural information is lost. Mathematically, it transforms the feature space $\mathbb{R}^d$ into a difference space $\mathbb{D}^d$. This approach prevents "structural collisions" where RKD might assign the same similarity score to two completely different geometric arrangements (e.g., points symmetric about an origin).
The Dual Distillation Network (DDNet)
Recognizing that base classes (with abundant data) and novel classes (with sparse data) exhibit different distribution characteristics, the authors propose the Dual Distillation Network (DDNet). This architecture utilizes a dual-branch strategy:
- Individual KD Branch: Preserves base class knowledge using standard distillation.
- DKD Branch: Captures the structural nuances of novel classes through displacement vectors.
To manage these branches during inference, an Instance-Aware Sample Selector dynamically weights the outputs. Trained with triplet loss, this selector determines whether a sample is more likely to belong to a base or novel class, bridging the semantic gap between well-established and newly introduced categories.
Key Insights and Results
Robustness through Decoupling
In traditional RKD, every sample is coupled via a similarity matrix, meaning a single outlier can "pollute" the entire gradient. In contrast, DKD's gradient analysis reveals that an outlier only affects $1/N$ of the teacher-student pairs. This makes the system significantly more resilient to noisy data.
Performance Benchmarks
DDNet was evaluated on three standard FSCIL benchmarks: CIFAR-100, miniImageNet, and CUB-200. The results demonstrate state-of-the-art performance:
- SOTA Accuracy: DDNet outperformed existing methods across all benchmarks, achieving an average 2.62% gain in Knowledge Retention (KR).
- Outlier Resilience: When subjected to data "attacks" (up to 20% outliers), DDNet reduced the average Accuracy Drop (AD) by 46.7% compared to competitors like BiDist.
- Versatility: The benefits of DKD extend beyond FSCIL; it also improves performance in general Class-Incremental Learning (CIL) and standard model compression tasks.
Conclusion
By shifting the focus from simple similarity to high-dimensional displacement, DKD provides a more stable and informative signal for knowledge distillation. While the Sample Selector remains a potential bottleneck for future refinement, the DDNet framework sets a new standard for balancing the retention of old knowledge with the acquisition of new, few-shot categories.
Summary updated