On Distilling the Displacement Knowledge for Few-Shot Class-Incremental Learning

Pengfei Fang, Yongchun Qin, Hui XuearXiv 2024 · DOI · arXiv

Introduction

Artificial intelligence systems often need to learn new categories over time from very few examples—a task known as Few-Shot Class-Incremental Learning (FSCIL). The primary hurdle in this domain is catastrophic forgetting: as a model learns new "novel" classes, it tends to overwrite the parameters necessary to recognize "base" classes it learned previously.

Existing solutions frequently employ Knowledge Distillation (KD) to align a "student" model's outputs with a "teacher" model (a previous iteration of itself). However, standard KD (Individual KD) often falters when data is scarce due to low-quality feature production. While Relational KD (RKD) attempts to address this by modeling inter-sample similarities, it frequently collapses complex structural information into a single dimension. This loss of directional data makes the model hypersensitive to outliers and prone to "structural collisions."

Displacement Knowledge Distillation (DKD)

To overcome the limitations of similarity-based distillation, this paper introduces Displacement Knowledge Distillation (DKD). Unlike RKD, which calculates a scalar similarity between samples, DKD calculates the displacement vector (the point-wise difference) between pairs of samples.

Comparison between Relational Knowledge Distillation (RKD) and Displacement Knowledge Distillation (DKD). RKD (c) collapses distinct geometric structures (a, b) into similar 1D representations, whereas DKD (d) preserves the structural differences.

By preserving both the distance and the angular (directional) relationship in the original feature space, DKD ensures that no structural information is lost. Mathematically, it transforms the feature space $\mathbb{R}^d$ into a difference space $\mathbb{D}^d$. This approach prevents "structural collisions" where RKD might assign the same similarity score to two completely different geometric arrangements (e.g., points symmetric about an origin).

The Dual Distillation Network (DDNet)

Recognizing that base classes (with abundant data) and novel classes (with sparse data) exhibit different distribution characteristics, the authors propose the Dual Distillation Network (DDNet). This architecture utilizes a dual-branch strategy:

  1. Individual KD Branch: Preserves base class knowledge using standard distillation.
  2. DKD Branch: Captures the structural nuances of novel classes through displacement vectors.

To manage these branches during inference, an Instance-Aware Sample Selector dynamically weights the outputs. Trained with triplet loss, this selector determines whether a sample is more likely to belong to a base or novel class, bridging the semantic gap between well-established and newly introduced categories.

Visualization of base and novel class distributions in the feature space. The DDNet architecture uses a dual-branch strategy to handle the distinct characteristics of these distributions.

Key Insights and Results

Robustness through Decoupling

In traditional RKD, every sample is coupled via a similarity matrix, meaning a single outlier can "pollute" the entire gradient. In contrast, DKD's gradient analysis reveals that an outlier only affects $1/N$ of the teacher-student pairs. This makes the system significantly more resilient to noisy data.

Performance Benchmarks

DDNet was evaluated on three standard FSCIL benchmarks: CIFAR-100, miniImageNet, and CUB-200. The results demonstrate state-of-the-art performance:

  • SOTA Accuracy: DDNet outperformed existing methods across all benchmarks, achieving an average 2.62% gain in Knowledge Retention (KR).
  • Outlier Resilience: When subjected to data "attacks" (up to 20% outliers), DDNet reduced the average Accuracy Drop (AD) by 46.7% compared to competitors like BiDist.
  • Versatility: The benefits of DKD extend beyond FSCIL; it also improves performance in general Class-Incremental Learning (CIL) and standard model compression tasks.

Conclusion

By shifting the focus from simple similarity to high-dimensional displacement, DKD provides a more stable and informative signal for knowledge distillation. While the Sample Selector remains a potential bottleneck for future refinement, the DDNet framework sets a new standard for balancing the retention of old knowledge with the acquisition of new, few-shot categories.

Summary updated

Abstract

Few-shot Class-Incremental Learning (FSCIL) addresses the challenges of evolving data distributions and the difficulty of data acquisition in real-world scenarios. To counteract the catastrophic forgetting typically encountered in FSCIL, knowledge distillation is employed as a way to maintain the knowledge from learned data distribution. Recognizing the limitations of generating discriminative feature representations in a few-shot context, our approach incorporates structural information between samples into knowledge distillation. This structural information serves as a remedy for the low quality of features. Diverging from traditional structured distillation methods that compute sample similarity, we introduce the Displacement Knowledge Distillation (DKD) method. DKD utilizes displacement rather than similarity between samples, incorporating both distance and angular information to significantly enhance the information density retained through knowledge distillation. Observing performance disparities in feature distribution between base and novel classes, we propose the Dual Distillation Network (DDNet). This network applies traditional knowledge distillation to base classes and DKD to novel classes, challenging the conventional integration of novel classes with base classes. Additionally, we implement an instance-aware sample selector during inference to dynamically adjust dual branch weights, thereby leveraging the complementary strengths of each approach. Extensive testing on three benchmarks demonstrates that DDNet achieves state-of-the-art results. Moreover, through rigorous experimentation and comparison, we establish the robustness and general applicability of our proposed DKD method.
BibTeX
@misc{fang2024distilling,
  author = {Pengfei Fang and Yongchun Qin and Hui Xue},
  title = {{On Distilling the Displacement Knowledge for Few-Shot Class-Incremental Learning}},
  year = {2024},
  eprint = {2412.11017},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.11017},
}
Figure