DKT: Diverse Knowledge Transfer Transformer for Class Incremental Learning

Xinyuan Gao, Yuhang He, Songlin Dong, Jie Cheng, Xing Wei, Yihong Gong

Abstract

Class Incremental Learning (CIL) requires models to learn new classes sequentially without forgetting previously acquired knowledge. This task is hindered by catastrophic forgetting and the stability-plasticity dilemma. While dynamic expandable networks offer a solution, they typically suffer from excessive parameter growth. We introduce the Diverse Knowledge Transfer Transformer (DKT), a framework that leverages specialized attention mechanisms and a compact "Token Pool" to transfer task-general and task-specific knowledge efficiently. DKT employs a duplex classifier to balance stability and plasticity, achieving state-of-the-art accuracy on benchmarks like CIFAR100 and ImageNet while maintaining a significantly smaller parameter footprint than existing expandable models.

Introduction

In Class Incremental Learning (CIL), models must adapt to new data streams while preserving old memories. This process is plagued by two main issues: Catastrophic Forgetting, where new weight updates overwrite old knowledge, and the Stability-Plasticity Dilemma, the challenge of being flexible enough for new information while remaining stable enough to retain the old.

Traditional approaches generally fall into two categories: regularization-based methods, which often lead to feature degradation, and dynamic expandable networks, which add new modules for every task. Although effective at preventing forgetting, expandable networks often face "parameter explosion," making them impractical for large-scale or resource-constrained applications.

Methodology: Diverse Knowledge Transfer

DKT addresses these challenges by utilizing the Transformer architecture's flexibility. Instead of duplicating entire feature extractors, it introduces a "Token Pool" and specialized attention blocks.

The DKT architecture featuring the Token Pool, GKAB, SKAB, and the Duplex Classifier.

Knowledge Transfer Mechanisms

  • GKAB (General Knowledge Transfer Attention Block): This block employs a task-general token updated across all tasks to capture broad, shared features, maintaining the model's foundational "general intelligence."
  • SKAB (Specific Knowledge Transfer Attention Block): For each new task, a tiny task-specific token (a 1 × 384 trainable vector) is initialized to capture unique task-related knowledge. This approach allows the model to grow with negligible parameter increases compared to full network expansion.
  • Duplex Classifier: To manage the stability-plasticity trade-off, DKT uses two heads: a frozen stability classifier for old categories and a trainable plasticity classifier for new ones. Their outputs are merged for final inference.

Optimization

The model is trained using a Cluster-Separation Loss, which encourages the clustering of similar categories in the feature space while maintaining separation between old and new task features, enhancing discriminability.

Experimental Evaluation

DKT was evaluated on standard benchmarks including CIFAR100, ImageNet100, and the full-scale ImageNet1000.

Performance and Efficiency

DKT consistently outperforms competitive baselines such as Dytox, WA, and DER. A key advantage is its parameter efficiency. On CIFAR100, DKT achieves higher accuracy than the previous state-of-the-art (DER) while utilizing only approximately 1/10th of the parameters.

Comparison of accuracy and parameter count across various CIL methods. DKT achieves high accuracy with a minimal parameter footprint.

On the 10-step CIFAR100 benchmark, DKT reached 75.83% average accuracy, significantly higher than Dytox (67.33%) and WA (69.46%). Its scalability was further demonstrated on ImageNet1000, where it maintained a top-1 average accuracy of 70.44%.

Conclusion and Future Work

DKT demonstrates that efficient class incremental learning can be achieved by focusing on token-based knowledge transfer rather than full backbone expansion. By separating general and specific knowledge, the model maintains high performance with linear complexity relative to the number of tasks.

Future research directions include reducing the dependency on a "memory buffer" (rehearsal) for old images, adapting the token-based mechanism to non-Transformer architectures, and addressing "blurry" incremental learning scenarios where task boundaries are not explicitly defined.

Summary updated

Abstract

In the context of incremental class learning, deep neural networks are prone to catastrophic forgetting, where the accuracy of old classes declines substantially as new knowledge is learned. While recent studies have sought to address this issue, most approaches suffer from either the stability-plasticity dilemma or excessive computational and parameter requirements. To tackle these challenges, we propose a novel framework, the Diverse Knowledge Transfer Transformer (DKT), which incorporates two knowledge transfer mechanisms that use attention mechanisms to transfer both task-specific and task-general knowledge to the current task, along with a duplex classifier to address the stability-plasticity dilemma. Additionally, we design a loss function that clusters similar categories and discriminates between old and new tasks in the feature space. The proposed method requires only a small number of extra parameters, which are negligible in comparison to the increasing number of tasks. We perform extensive experiments on CIFAR100, ImageNet100, and ImageNet1000 datasets, which demonstrate that our method outperforms other competitive methods and achieves state-of-the-art performance. Our source code is available at https://github.com/MIVXJTU/DKT.
BibTeX
@inproceedings{gao2023dkt,
  author = {Xinyuan Gao and Yuhang He and Songlin Dong and Jie Cheng and Xing Wei and Yihong Gong},
  title = {{DKT: Diverse Knowledge Transfer Transformer for Class Incremental Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.02321},
}
Figure