Abstract
Incremental learning aims to train models on a sequence of tasks without forgetting previous knowledge, a phenomenon known as catastrophic forgetting. Traditional knowledge distillation (KD) methods often fail because they rely on overconfident point estimates from teacher models. This paper proposes a novel framework that distills both model uncertainty and spatial attention. By incorporating aleatoric uncertainty distillation and multi-stage self-attention distillation, the model preserves the "probabilistic signature" and discriminative regions of old tasks. Experimental results on CIFAR-100 and ImageNet-100 demonstrate that the proposed method significantly reduces forgetting and maintains higher accuracy compared to state-of-the-art baselines.
Introduction
Deep neural networks excel at learning from static datasets but struggle with sequential tasks. When trained on new data, the weights optimized for previous tasks are often overwritten, leading to a sharp decline in performance on original tasks—a challenge termed catastrophic forgetting. While Knowledge Distillation (KD) is a common mitigation strategy, standard implementations typically match only the final softmax outputs. These outputs often lack the nuance of the teacher's internal state, particularly its uncertainty and the specific features it prioritized. This work addresses these gaps by introducing uncertainty and attention as key components of the distillation process.
Proposed Method
The proposed architecture extends the Learning without Forgetting (LwF) framework by introducing two primary mechanisms to stabilize knowledge transfer across incremental steps. The system incorporates a weight-freeze network to represent the teacher's knowledge and applies distillation across multiple stages of the feature extractor.
1. Uncertainty-Based Distillation
The framework estimates both epistemic (model) uncertainty via Monte Carlo Dropout and aleatoric (data) uncertainty through a dedicated variance network. An Aleatoric Distillation Loss ($L_A$) is employed to ensure the student model's uncertainty matches that of the teacher. This serves as a "probabilistic signature" of the old data distribution, preventing the model from becoming overconfident in regions where it was previously uncertain.
2. Multi-Stage Self-Attention Distillation
To prevent semantic drift, Self-Attention modules are integrated at multiple stages of the feature extractor (e.g., after different blocks in a ResNet). An Attention Distillation Loss ($L_m$) forces the current model to maintain focus on the same discriminative regions—such as the shape of a bird or the wheels of a car—that were critical for previous tasks. This ensures that the fundamental feature representations remain consistent as new classes are added.
Experimental Results
The method was evaluated on CIFAR-100 and ImageNet-100 using various incremental protocols (5, 10, and 20 class increments). The proposed approach, dubbed Ours-AU (Attention-Uncertainty), consistently outperformed several state-of-the-art baselines in both accuracy and forgetting metrics.
On CIFAR-100 with 10-class increments, the model achieved 65.4% accuracy, a significant improvement over the 56.8% baseline. Furthermore, the forgetting rate was markedly reduced; for instance, in the 20-class increment task on ImageNet-100, the forgetting rate was limited to 8.1%, compared to 18.4% for the baseline. Qualitative analysis via attention maps confirmed that the model retains its ability to focus on relevant object features even after multiple rounds of incremental training.
Conclusion
By distilling uncertainty and spatial attention, this research provides a more robust shield against catastrophic forgetting. The results suggest that the "nuance" of old knowledge—what the model was unsure about and where it looked—is just as important as the final prediction. While the computational cost of uncertainty estimation via multiple forward passes remains a consideration, the gains in stability and performance make it a compelling direction for lifelong learning systems. Future work will explore scaling this approach to larger datasets like the full ImageNet-1k and investigating its applicability to non-classification tasks.
Summary updated