Multi-Centroid Task Descriptor for Dynamic Class Incremental Inference

Tenghao Cai, Zhizhong Zhang, Xin Tan, Yanyun Qu, Guannan Jiang, Chengjie Wang, Yuan Xie

Abstract

Class-Incremental Learning (Class-IL) aims to learn new concepts sequentially without forgetting previous ones, a task significantly more difficult than Task-Incremental Learning (Task-IL) where task identifiers are provided at test time. This paper introduces a Dynamic Inference Network that utilizes a multi-centroid task descriptor to accurately predict task IDs during inference. By employing a gate network trained with bipartite matching and knowledge distillation, the proposed method bridges the performance gap between Class-IL and Task-IL, achieving state-of-the-art accuracy on large-scale benchmarks including ImageNet-1000.

Introduction

Incremental learning (IL) is essential for developing AI systems that can adapt to evolving data streams. While Task-IL provides the model with task context during testing, Class-IL requires the model to distinguish between all learned classes without such priors. The authors identify a significant "performance gap" where even state-of-the-art Class-IL models perform substantially better if provided with the task ID. The challenge in bridging this gap lies in the arbitrary nature of task groupings, which often lack semantic cohesion, making single-prototype task descriptors ineffective for capturing the high variance of unrelated classes within a single task.

Methodology

The proposed solution is a Dynamic Inference Network featuring a standalone gate network designed to predict task IDs and route inputs to the most relevant model branches.

Multi-Centroid Task Descriptor

Recognizing that classes within a task may be semantically diverse, the authors represent each task using multiple trainable centroids rather than a single point. This allows the descriptor to capture the disjoint clusters inherent in task-based data, effectively modeling the "disjointed" nature of arbitrary task groupings.

Bipartite Matching and Training

To ensure centroids remain representative and diverse, the Hungarian Algorithm is used during training to match class prototypes to specific centroids. This prevents centroid collapse and ensures full coverage of the task's semantic space. The training proceeds in two stages:

  1. Supernet Training: A multi-branch backbone (using DER as a baseline) is trained to learn features.
  2. Gate Training: The gate network is trained independently. To overcome the lack of old-task data, the authors use RandAugment for data synthesis and Knowledge Distillation to inherit semantic knowledge from the already-trained supernet.
Overview of the training process with multi-centroid task descriptors and the dynamic inference mechanism.

Adaptive Inference

During inference, the gate network computes similarities between input features and task centroids. These similarities weight the outputs of the corresponding task branches. This mechanism not only improves accuracy but also allows for efficiency gains; by activating only the "Top-N" most likely branches, inference time can be reduced to just 10% of the baseline with negligible accuracy loss.

Experiments and Results

The method was validated on standard benchmarks including CIFAR-100, ImageNet-100, and the full-scale ImageNet-1000 dataset.

Key Results

The approach consistently outperformed existing baselines. On CIFAR100-B0S50 (50 steps), it achieved an average accuracy of 72.41%, a 3.40% improvement over the DER baseline. On the 1,000-class ImageNet challenge, it reached a top-1 average accuracy of 70.80%, surpassing competitive models like DyTox and FOSTER.

Accuracy comparison against state-of-the-art methods on CIFAR100-B0 with 20 incremental steps.

Ablation studies confirmed that the multi-centroid approach provides a ~1.5% boost over simple flat classification, validating the importance of the cluster-based descriptor.

Conclusion

By treating task identification as a dynamic routing problem supported by multi-centroid descriptors, this work effectively narrows the gap between Class-IL and Task-IL. While the gate network introduces a parameter overhead (approximately 11.2M parameters for ResNet-18), the gains in accuracy and inference efficiency make it a robust solution for large-scale incremental learning. Future research may explore merging similar branches to reduce the overall parameter footprint as the model grows.

Summary updated

Abstract

Incremental learning could be roughly divided into two categories, i.e., class- and task-incremental learning. The main difference is whether the task ID is given during evaluation. In this paper, we show this task information is indeed a strong prior knowledge, which will bring significant improvement over class-incremental learning baseline, e.g., DER [39]. Based on this observation, we propose a gate network to predict the task ID for class incremental inference. This is challenging as there is no explicit semantic relationship between categories in the concept of task. Therefore, we propose a multi-centroid task descriptor by assuming the data within a task can form multiple clusters. The cluster centers are optimized by pulling relevant sample-centroid pairs while pushing others away, which ensures that there is at least one centroid close to a given sample. To select relevant pairs, we use class prototypes as proxies and solve a bipartite matching problem, making the task descriptor representative yet not degenerate to uni-modal. As a result, our dynamic inference network is trained independently of baseline and provides a flexible, efficient solution to distinguish between tasks. Extensive experiments show our approach achieves state-of-the-art results, e.g., we achieve 72.41% average accuracy on CIFAR100-BOS50, outperforming DER by 3.40%.
BibTeX
@inproceedings{cai2023multi,
  author = {Tenghao Cai and Zhizhong Zhang and Xin Tan and Yanyun Qu and Guannan Jiang and Chengjie Wang and Yuan Xie},
  title = {{Multi-Centroid Task Descriptor for Dynamic Class Incremental Inference}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.00705},
}
Figure