Resolving Task Confusion in Dynamic Expansion Architectures for Class Incremental Learning

Bingchen Huang, Zhineng Chen, Peng Zhou, Jiayin Chen, Zuxuan Wu

Introduction

Class Incremental Learning (CIL) is a fundamental challenge in computer vision, requiring models to learn new classes sequentially without suffering from "catastrophic forgetting." A prominent solution is the Dynamic Expansion Architecture (DEA), which freezes parameters of old tasks and adds new sub-networks for incoming data. While DEAs effectively preserve old features, they introduce a critical bottleneck: Task Confusion.

The Challenge: Inter-Task and Old-New Confusion

The authors identify two primary forms of confusion that degrade performance in expanding networks:

  1. Inter-Task Confusion (ITC): The model learns to distinguish classes within individual tasks but fails to learn the decision boundaries between tasks. As a result, it may confuse a class from Task A with one from Task B.
  2. Old-New Confusion (ONC): A structural bias where the final classification layer develops larger weight magnitudes for recently learned classes. This statistical imbalance causes the model to "prefer" new labels over old ones during inference.
Figure 2: (a) Conceptual illustration of Inter-Task Confusion (ITC). (b) Visualization of Old-New Confusion (ONC) through L2 norms of weights across class indices. (c) Error statistics comparing the baseline, Classifier Re-scoring (CR), and the proposed TCIL method.

The TCIL Framework

To bridge the gap between isolated sub-networks, the paper introduces Task Correlated Incremental Learning (TCIL). This framework resolves the "seam" problem between tasks through three integrated components:

Multi-level Knowledge Distillation (MLKD)

Unlike traditional distillation used for model compression, TCIL uses it to "correlate" independent sub-networks. It applies an $L_2$ loss at the feature level to align new extractor outputs with old ones and a KL-divergence loss at the logit level to maintain prediction consistency across the entire class spectrum.

Feature Fusion Module (FFM)

Since sub-networks are trained at different times, their features may not be naturally compatible. TCIL employs a dual-attention mechanism (channel and spatial attention) to refine and weight the importance of features from all tasks before they reach the final classifier.

Classifier Re-scoring (CR)

To mitigate the weight magnitude bias (ONC), TCIL calculates the mean weight norms of old versus new classes. It then applies a scaling coefficient ($\gamma$) to the new class logits during inference, mathematically leveling the playing field between old and new knowledge.

For resource-constrained environments, the authors also propose TCIL-Lite, which utilizes kernel pruning based on geometric medians to reduce the parameter count by nearly 4x with minimal impact on accuracy.

Key Results and Performance

TCIL was evaluated on CIFAR100 and ImageNet100 benchmarks, demonstrating superior performance in both rehearsal (with memory buffer) and non-rehearsal (zero old data) settings:

  • State-of-the-Art Accuracy: On CIFAR100 (10 steps), TCIL achieved 77.3% average accuracy, outperforming leading methods like DER (75.4%) and DyTox (73.7%).
  • Superiority in Non-Rehearsal: In the challenging zero-memory scenario, TCIL reached 39.9% "Last" accuracy on CIFAR100, a massive improvement over the previous state-of-the-art, LwM (26.1%).
  • Efficiency: TCIL-Lite maintained high performance (76.7% vs 77.3% on CIFAR100) while significantly reducing the model's memory footprint.

Conclusion

TCIL demonstrates that simply freezing old weights is insufficient for robust incremental learning. By treating distillation as a correlation tool and correcting structural biases in the classifier, TCIL effectively resolves task confusion. This approach sets a new standard for expanding architectures, particularly in privacy-sensitive or storage-limited applications where storing old data for rehearsal is not an option.

Summary updated

Abstract

The dynamic expansion architecture is becoming popular in class incremental learning, mainly due to its advantages in alleviating catastrophic forgetting. However, task confu- sion is not well assessed within this framework, e.g., the discrepancy between classes of different tasks is not well learned (i.e., inter-task confusion, ITC), and certain prior- ity is still given to the latest class batch (i.e., old-new con- fusion, ONC). We empirically validate the side effects of the two types of confusion. Meanwhile, a novel solution called Task Correlated Incremental Learning (TCIL) is pro- posed to encourage discriminative and fair feature utilization across tasks. TCIL performs a multi-level knowledge distil- lation to propagate knowledge learned from old tasks to the new one. It establishes information flow paths at both fea- ture and logit levels, enabling the learning to be aware of old classes. Besides, attention mechanism and classifier re- scoring are applied to generate more fair classification scores. We conduct extensive experiments on CIFAR100 and Ima- geNet100 datasets. The results demonstrate that TCIL con- sistently achieves state-of-the-art accuracy. It mitigates both ITC and ONC, while showing advantages in battle with catas- trophic forgetting even no rehearsal memory is reserved. Source code: https://github.com/YellowPancake/TCIL.
BibTeX
@inproceedings{huang2023resolving,
  author = {Bingchen Huang and Zhineng Chen and Peng Zhou and Jiayin Chen and Zuxuan Wu},
  title = {{Resolving Task Confusion in Dynamic Expansion Architectures for Class Incremental Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i1.25170},
}
Figure