Long-Tail Class Incremental Learning via Independent SUb-Prototype Construction

Xi Wang, Xu Yang, Jie Yin, Kun Wei, Cheng DengCVPR 2024 · OpenReview

Abstract

Long-tail class incremental learning (LT-CIL) is designed to perpetually acquire novel knowledge from an imbalanced and perpetually evolving data stream while ensuring the retention of previously acquired knowledge. The existing method only re-balances data distribution and ignores exploring the potential relationship between different samples, causing non-robust representations and even severe forgetting in classes with few samples. In this paper, we constructed two parallel spaces simultaneously: 1) Sub-prototype space and 2) Reminiscence space to learn robust representations while alleviating forgetfulness. Concretely, we advance the concept of the sub-prototype space, which amalgamates insights from diverse classes. This integration facilitates the mutual complementarity of varied knowledge, thereby augmenting the attainment of more robust representations. Furthermore, we introduce the reminiscence space, which encapsulates each class distribution, aiming to constraint model optimization and mitigate the phenomenon of forgetting. The tandem utilization of the two parallel spaces effectively alleviates the adverse consequences associated with imbalanced data distribution, preventing forgetting without needing replay examples. Extensive experiments demonstrate that our method achieves state-of-the-art performance on various benchmarks.

Introduction

Deep learning models deployed in real-world environments face two primary challenges: Catastrophic Forgetting and Long-Tail Distributions. While standard incremental learning addresses the former—the tendency of models to "overwrite" old knowledge when learning new tasks—Long-Tail Class Incremental Learning (LT-CIL) introduces the added complexity of data imbalance. In LT-CIL, models must contend with both intra-task imbalance (skewed distributions within a single learning stage) and inter-task imbalance (disparities between old and new data).

Existing approaches often focus solely on re-balancing data distributions, neglecting the underlying semantic relationships between classes. This leads to fragile representations for rare categories. This paper proposes that minority classes can benefit from "knowledge amalgamation." By decomposing classes into shared structural features—sub-prototypes—rare classes can leverage robust features learned from majority classes.

Conceptual overview of the Sub-prototype Place where common features are shared between majority and minority classes.

Methodology

The proposed framework utilizes two parallel modules to create robust, forget-proof representations without the need for storing raw image exemplars.

Sub-prototype Space (SS)

Rather than representing each class as a monolithic prototype, the model learns a set of independent basis vectors, or sub-prototypes. This approach recognizes that many classes share common attributes (e.g., "eyes," "paws," or "tails"). By learning these shared components, the model uses the rich information available in majority classes to bolster the representation of minority classes that share similar features. To ensure a balanced feature space, a "controlling factor" is implemented to encourage the selection of under-utilized basis vectors. During training, features are re-sampled from this space to balance the training batch dynamically.

Reminiscence Space

To mitigate forgetting without the privacy and storage overhead of image replay, the authors introduce a distribution-based memory. For each completed task, the model stores the statistical anchors—specifically the feature mean and covariance matrix—for each class. During subsequent tasks, these stored distributions are used to apply a distillation loss. This "reminds" the model of the feature-space characteristics of previous classes, preventing the sub-prototype space from shifting drastically as new knowledge is integrated.

The proposed framework featuring the Sub-prototype Space and Reminiscence Space.

Experimental Results

The framework was evaluated on CIFAR100 and ImageNet-Subset under two scenarios: Ordered LT-CIL (decreasing sample counts over tasks) and Shuffled LT-CIL (randomized imbalance).

The method achieved State-of-the-Art (SOTA) performance across all benchmarks. On CIFAR100 (Ordered, 5 tasks), it reached 44.8% accuracy, a 2.5% improvement over previous benchmarks like LUCIR+LWS. Notably, the model demonstrated significantly higher stability in minority class accuracy compared to baseline methods like IL2A and BalPoE. Heatmap analysis further confirmed that internal representations remained consistent over time, indicating high resistance to catastrophic forgetting.

Conclusion

This research demonstrates that decomposing class representations into shared sub-prototypes and maintaining statistical reminiscence of past distributions effectively addresses the dual challenges of LT-CIL. While the method introduces some computational overhead during training, it eliminates the need for raw data storage and provides a robust path for perpetual learning in imbalanced environments. Future work may explore automated methods for determining the optimal number of sub-prototypes for diverse datasets.

Summary updated

Abstract

Long-tail class incremental learning (LT-CIL) is designed to perpetually acquire novel knowledge from an imbalanced and perpetually evolving data stream while ensuring the retention of previously acquired knowledge. The existing method only re-balances data distribution and ignores exploring the potential relationship between different samples, causing non-robust representations and even severe forgetting in classes with few samples. In this paper, we constructed two parallel spaces simultaneously: 1) Sub-prototype space and 2) Reminiscence space to learn robust representations while alleviating forgetfulness. Concretely, we advance the concept of the sub-prototype space, which amalgamates insights from diverse classes. This integration facilitates the mutual complementarity of varied knowledge, thereby augmenting the attainment of more robust representations. Furthermore, we introduce the reminiscence space, which encapsulates each class distribution, aiming to costraint model optimization and mitigate the phenomenon of forgetting. The tandem utilization of the two parallel spaces effectively alleviates the adverse consequences associated with imbalanced data distribution, preventing forgetting without needing replay examples. Extensive experiments demonstrate that our method achieves state-of-the-art performance on various benchmarks.
BibTeX
@inproceedings{wang2024long,
  author = {Xi Wang and Xu Yang and Jie Yin and Kun Wei and Cheng Deng},
  title = {{Long-Tail Class Incremental Learning via Independent SUb-Prototype Construction}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  url = {https://openreview.net/forum?id=f1H9nxb5Dx},
}
Figure