Learning from Students: Online Contrastive Distillation Network for General Continual Learning

Jin Li, Zhong Ji, Gang Wang, Qiang Wang, Feng GaoIJCAI 2022 · DOI

TL;DR

This paper introduces OCD-Net, a novel approach to General Continual Learning (GCL) that addresses catastrophic forgetting by using the "student" model to guide the training of the "teacher" model in an online fashion. This allows the teacher to accumulate knowledge from the student, leading to improved performance and reduced bias compared to traditional offline distillation methods.

What Problem Does This Paper Address?

Continual Learning (CL) aims to enable machine learning models to learn new information sequentially without forgetting previously acquired knowledge. A major hurdle is "catastrophic forgetting," where learning a new task drastically degrades performance on older tasks. General Continual Learning (GCL) further complicates this by assuming no explicit task boundaries and a bounded memory. While knowledge distillation (KD) is a popular technique to transfer knowledge, traditional offline KD methods suffer from a "response bias" inherited from the old teacher model, limiting their effectiveness. This paper tackles the challenge of building a robust GCL system that overcomes this bias and effectively preserves and accumulates knowledge.

Methods

The proposed OCD-Net introduces two key modules:

  • Online Response Distillation Module (ORD-Module): Instead of using a fixed, old teacher model, OCD-Net employs an "online teacher" that is continuously updated by integrating the parameters of the current student model. This allows the teacher to benefit from the student's latest knowledge and reduce the response bias. The teacher is updated using a "random momentum update" approach, where student parameters are randomly incorporated.
  • Contrastive Relation Distillation Module (CRD-Module): To further preserve learned knowledge, this module focuses on maintaining the consistency of class similarities between the teacher and student models. It uses embedding spaces to ensure that samples from the same class are pulled closer together, while samples from different classes are pushed apart.
  • Adaptive Perception: An adaptive mechanism is introduced to weigh the importance of different teacher responses during distillation. This allows the student model to focus more on high-quality, less biased responses from the teacher.

Datasets

The paper evaluates OCD-Net on six benchmark datasets:

  • Split Benchmarks:
    • Split CIFAR-10 (S-CIFAR-10)
    • Split CIFAR-100 (S-CIFAR-100)
    • Split Tiny-ImageNet (S-Tiny-ImageNet)
  • Smooth Benchmarks:
    • MNIST-360
    • Generalized CIFAR-100 (UG-CIFAR-100)
    • Longtail Generalized CIFAR-100 (LG-CIFAR-100)

Key Insights and Ideas

  • Student-Guided Teacher: The core innovation is using the student model to train the teacher model. This "learning from students" paradigm allows the teacher to accumulate knowledge more effectively and with less bias than traditional methods.
  • Mitigating Response Bias: By updating the teacher online with student parameters, the inherited bias from older models is significantly reduced, leading to more reliable knowledge transfer.
  • Complementary Distillation: The combination of response distillation (ORD-Module) and relation distillation (CRD-Module) provides a more comprehensive approach to preserving knowledge, addressing both individual sample responses and the underlying class structure.
  • Adaptive Focus: The adaptive perception mechanism intelligently weights teacher responses, ensuring that the student model learns from the most informative signals.

Key Results

OCD-Net consistently outperforms state-of-the-art approaches across various buffer sizes on both split and smooth benchmarks. For instance, it achieves significant improvements, ranging from 3.03% to 8.71% over existing methods. Ablation studies confirm the effectiveness of each component (ORD-Module, CRD-Module, and adaptive perception), demonstrating their complementary roles in mitigating catastrophic forgetting. The paper also shows that OCD-Net alleviates the classifier bias observed in traditional offline distillation methods.

Limitations and Future Work

The authors acknowledge that the performance on the LG-CIFAR-100 dataset is worse than on UG-CIFAR-100, suggesting that class imbalance exacerbates catastrophic forgetting. Future work is suggested to extend OCD-Net to more challenging scenarios such as semi-supervised GCL and non-exemplar GCL.

Summary updated

BibTeX
@inproceedings{li2022learning,
  author = {Jin Li and Zhong Ji and Gang Wang and Qiang Wang and Feng Gao},
  title = {{Learning from Students: Online Contrastive Distillation Network for General Continual Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/446},
}
Figure