Abstract
Class Incremental Learning (CIL) requires models to learn new classes sequentially without forgetting previously acquired knowledge. This task is hindered by catastrophic forgetting and the stability-plasticity dilemma. While dynamic expandable networks offer a solution, they typically suffer from excessive parameter growth. We introduce the Diverse Knowledge Transfer Transformer (DKT), a framework that leverages specialized attention mechanisms and a compact "Token Pool" to transfer task-general and task-specific knowledge efficiently. DKT employs a duplex classifier to balance stability and plasticity, achieving state-of-the-art accuracy on benchmarks like CIFAR100 and ImageNet while maintaining a significantly smaller parameter footprint than existing expandable models.
Introduction
In Class Incremental Learning (CIL), models must adapt to new data streams while preserving old memories. This process is plagued by two main issues: Catastrophic Forgetting, where new weight updates overwrite old knowledge, and the Stability-Plasticity Dilemma, the challenge of being flexible enough for new information while remaining stable enough to retain the old.
Traditional approaches generally fall into two categories: regularization-based methods, which often lead to feature degradation, and dynamic expandable networks, which add new modules for every task. Although effective at preventing forgetting, expandable networks often face "parameter explosion," making them impractical for large-scale or resource-constrained applications.
Methodology: Diverse Knowledge Transfer
DKT addresses these challenges by utilizing the Transformer architecture's flexibility. Instead of duplicating entire feature extractors, it introduces a "Token Pool" and specialized attention blocks.
Knowledge Transfer Mechanisms
- GKAB (General Knowledge Transfer Attention Block): This block employs a task-general token updated across all tasks to capture broad, shared features, maintaining the model's foundational "general intelligence."
- SKAB (Specific Knowledge Transfer Attention Block): For each new task, a tiny task-specific token (a 1 × 384 trainable vector) is initialized to capture unique task-related knowledge. This approach allows the model to grow with negligible parameter increases compared to full network expansion.
- Duplex Classifier: To manage the stability-plasticity trade-off, DKT uses two heads: a frozen stability classifier for old categories and a trainable plasticity classifier for new ones. Their outputs are merged for final inference.
Optimization
The model is trained using a Cluster-Separation Loss, which encourages the clustering of similar categories in the feature space while maintaining separation between old and new task features, enhancing discriminability.
Experimental Evaluation
DKT was evaluated on standard benchmarks including CIFAR100, ImageNet100, and the full-scale ImageNet1000.
Performance and Efficiency
DKT consistently outperforms competitive baselines such as Dytox, WA, and DER. A key advantage is its parameter efficiency. On CIFAR100, DKT achieves higher accuracy than the previous state-of-the-art (DER) while utilizing only approximately 1/10th of the parameters.
On the 10-step CIFAR100 benchmark, DKT reached 75.83% average accuracy, significantly higher than Dytox (67.33%) and WA (69.46%). Its scalability was further demonstrated on ImageNet1000, where it maintained a top-1 average accuracy of 70.44%.
Conclusion and Future Work
DKT demonstrates that efficient class incremental learning can be achieved by focusing on token-based knowledge transfer rather than full backbone expansion. By separating general and specific knowledge, the model maintains high performance with linear complexity relative to the number of tasks.
Future research directions include reducing the dependency on a "memory buffer" (rehearsal) for old images, adapting the token-based mechanism to non-Transformer architectures, and addressing "blurry" incremental learning scenarios where task boundaries are not explicitly defined.
Summary updated