Introduction
Class Incremental Learning (CIL) is a fundamental challenge in computer vision, requiring models to learn new classes sequentially without suffering from "catastrophic forgetting." A prominent solution is the Dynamic Expansion Architecture (DEA), which freezes parameters of old tasks and adds new sub-networks for incoming data. While DEAs effectively preserve old features, they introduce a critical bottleneck: Task Confusion.
The Challenge: Inter-Task and Old-New Confusion
The authors identify two primary forms of confusion that degrade performance in expanding networks:
- Inter-Task Confusion (ITC): The model learns to distinguish classes within individual tasks but fails to learn the decision boundaries between tasks. As a result, it may confuse a class from Task A with one from Task B.
- Old-New Confusion (ONC): A structural bias where the final classification layer develops larger weight magnitudes for recently learned classes. This statistical imbalance causes the model to "prefer" new labels over old ones during inference.
The TCIL Framework
To bridge the gap between isolated sub-networks, the paper introduces Task Correlated Incremental Learning (TCIL). This framework resolves the "seam" problem between tasks through three integrated components:
Multi-level Knowledge Distillation (MLKD)
Unlike traditional distillation used for model compression, TCIL uses it to "correlate" independent sub-networks. It applies an $L_2$ loss at the feature level to align new extractor outputs with old ones and a KL-divergence loss at the logit level to maintain prediction consistency across the entire class spectrum.
Feature Fusion Module (FFM)
Since sub-networks are trained at different times, their features may not be naturally compatible. TCIL employs a dual-attention mechanism (channel and spatial attention) to refine and weight the importance of features from all tasks before they reach the final classifier.
Classifier Re-scoring (CR)
To mitigate the weight magnitude bias (ONC), TCIL calculates the mean weight norms of old versus new classes. It then applies a scaling coefficient ($\gamma$) to the new class logits during inference, mathematically leveling the playing field between old and new knowledge.
For resource-constrained environments, the authors also propose TCIL-Lite, which utilizes kernel pruning based on geometric medians to reduce the parameter count by nearly 4x with minimal impact on accuracy.
Key Results and Performance
TCIL was evaluated on CIFAR100 and ImageNet100 benchmarks, demonstrating superior performance in both rehearsal (with memory buffer) and non-rehearsal (zero old data) settings:
- State-of-the-Art Accuracy: On CIFAR100 (10 steps), TCIL achieved 77.3% average accuracy, outperforming leading methods like DER (75.4%) and DyTox (73.7%).
- Superiority in Non-Rehearsal: In the challenging zero-memory scenario, TCIL reached 39.9% "Last" accuracy on CIFAR100, a massive improvement over the previous state-of-the-art, LwM (26.1%).
- Efficiency: TCIL-Lite maintained high performance (76.7% vs 77.3% on CIFAR100) while significantly reducing the model's memory footprint.
Conclusion
TCIL demonstrates that simply freezing old weights is insufficient for robust incremental learning. By treating distillation as a correlation tool and correcting structural biases in the classifier, TCIL effectively resolves task confusion. This approach sets a new standard for expanding architectures, particularly in privacy-sensitive or storage-limited applications where storing old data for rehearsal is not an option.
Summary updated