Abstract
Dynamic Expansion Architectures (DEAs) have emerged as a powerful paradigm for Class Incremental Learning (CIL) due to their ability to mitigate catastrophic forgetting by freezing old parameters and adding new sub-networks. However, these architectures suffer from "task confusion," where the model fails to distinguish between classes learned in different tasks. This paper introduces Task Correlated Incremental Learning (TCIL), a novel framework that addresses two primary forms of confusion: Inter-Task Confusion (ITC) and Old-New Confusion (ONC). By employing multi-level knowledge distillation, attention-based feature fusion, and classifier re-scoring, TCIL achieves state-of-the-art performance on CIFAR100 and ImageNet100, demonstrating exceptional robustness even in the absence of rehearsal memory.
Introduction
In the field of Class Incremental Learning (CIL), the primary challenge is to learn a sequence of tasks without forgetting previous knowledge. Dynamic Expansion Architectures (DEAs) address this by expanding the network for each new task while keeping previous parameters fixed. While this effectively prevents the erasure of old features, it introduces a new problem: the model lacks a global perspective, leading to significant confusion when classifying images across different task boundaries.
Analyzing Task Confusion
The authors identify two distinct types of confusion inherent in DEAs:
- Inter-Task Confusion (ITC): Since sub-networks are trained independently on specific tasks, the model does not learn the discriminative features necessary to distinguish between similar classes across different tasks (e.g., distinguishing a "fish" in Task A from a "fish" in Task B).
- Old-New Confusion (ONC): This is a structural bias where the final classification layer develops a "recency bias." The weights associated with the newest classes tend to have significantly larger L2 norms than the frozen weights of older classes, causing the model to disproportionately predict the most recently learned categories.
The TCIL Framework
To resolve these issues, the proposed Task Correlated Incremental Learning (TCIL) framework introduces several key innovations:
Multi-level Knowledge Distillation (MLKD)
TCIL uses distillation to "correlate" independent sub-networks. At the feature level, it prevents the new feature extractor from drifting too far from the established feature space. At the logit level, it employs KL divergence to ensure the joint classifier preserves the prediction logic of earlier iterations, effectively creating an information flow between tasks.
Feature Fusion Module (FFM)
Rather than simple concatenation, TCIL utilizes a Feature Fusion Module equipped with spatial and channel attention mechanisms. This allows the model to dynamically weigh and integrate features from various sub-networks, enhancing the discriminative power of the combined representation.
Classifier Re-scoring (CR)
To mitigate ONC, TCIL applies a post-training normalization step. By calculating the mean norm of weights for old versus new classes, the model applies a scaling factor to the new class logits. This "re-scoring" ensures a level playing field, correcting the structural recency bias.
TCIL-Lite
For resource-constrained environments, the authors introduce TCIL-Lite. This version utilizes geometric median-based pruning to remove redundant filters within the expanding sub-networks, significantly reducing parameter growth with minimal impact on accuracy.
Experimental Results
TCIL was evaluated on CIFAR100 and ImageNet100 across various incremental settings, showing dominance in both rehearsal and non-rehearsal scenarios.
- Accuracy: On CIFAR100 (10 steps), TCIL achieved an average accuracy of 77.30%, surpassing the previous state-of-the-art DEA, DER (75.36%).
- Non-Rehearsal Performance: In the most challenging "zero memory" setting, TCIL demonstrated remarkable stability. On CIFAR100 (10 steps), it maintained a final accuracy of 39.91%, whereas DER dropped to 8.88%.
- Efficiency: TCIL-Lite successfully reduced the parameter count on ImageNet100 from 64.1M to 14.5M while maintaining accuracy within 0.2% of the full model.
Conclusion
TCIL effectively addresses the "task confusion" bottleneck in dynamic expansion architectures. By fostering correlation between tasks through distillation and attention, and correcting structural biases through re-scoring, it provides a robust and efficient solution for class incremental learning. Future work will explore deeper parameter sharing and applications in non-visual domains.
Summary updated