A Tale of Two CILs: The Connections Between Class Incremental Learning and Class Imbalanced Learning, and Beyond

Chen He, Ruiping Wang, Xilin ChenWorkshops @ CVPR 2021 · OpenReview

Abstract: Catastrophic forgetting, the main challenge of Class Incremental Learning (CIncL), is closely related to the classifier's bias due to imbalanced data. Most researchers resort to empirical techniques to remove this bias, yet these "anti-bias" tricks share many ideas with the field of Class Imbalanced Learning (CImbL). This paper comprehensively analyzes the connections between these two fields. We provide a panoramic view of recent bias correction tricks from the perspective of handling class imbalance and demonstrate that an adapted post-scaling technique from CImbL is on par with or even outperforms state-of-the-art CIncL methods. Visualization via violin plots and polar charts further sheds light on how these methods address class imbalance from a geometric perspective. Our findings raise the question of whether CIncL is essentially degenerating into a class imbalance problem.

Introduction: Are We Reinventing the Wheel?

In Class Incremental Learning (CIncL), a model must learn new categories in stages without forgetting old ones. Due to memory constraints, we can only store a small fraction of old data (exemplars), leading to a training set that is massively biased toward the newest classes. This imbalance is a primary driver of catastrophic forgetting, where the model’s accuracy on previously learned classes plummets.

While the CIncL community has developed numerous "anti-bias" tricks to mitigate this, these methods are often empirical and disconnected from the established field of Class Imbalanced Learning (CImbL). This research seeks to bridge that gap, asking if CIncL is simply a specific, severe instance of the class imbalance problem.

Methodology: Mapping the Connection

The authors first establish a systematic mapping between popular CIncL methods and classic CImbL strategies. As shown in the table below, many "innovations" in incremental learning have direct counterparts in the imbalance literature, such as under-sampling, two-phase training, and probability calibration.

Table 1: Mapping CIncL methods to their corresponding CImbL concepts and techniques.

The Post-Scaling Technique

Building on this connection, the authors introduce a "post-scaling" layer derived from Bayesian theory. Instead of complex training routines, they apply a logit-level bias term at test time:

$$\log\left(\frac{p_{ts}(c)}{p_{tr}(c)}\right)$$

This adjusts the model's predictions based on the ratio of samples seen during training ($p_{tr}$) versus the expected distribution at test time ($p_{ts}$). This simple adjustment requires virtually zero training overhead yet addresses the core imbalance issue directly.

Geometric Insights into Bias

To understand why these techniques work, the paper employs geometric visualizations. By using polar charts to visualize the angular distribution of features, the researchers demonstrate that classifier bias is not just a numerical error but a physical shift in decision boundaries.

Figure 1: Polar charts visualizing the angular distribution of features and decision boundaries (red lines) for different classes.

In an imbalanced scenario, the decision boundary naturally "shrinks" around the minority (old) classes. Techniques like post-scaling and weight aligning effectively "push" or "rotate" these boundaries to restore a fair volume for the minority classes in the feature space.

Experimental Results

The study evaluated these methods on standard benchmarks including CIFAR-100 and a 100-class subset of ImageNet. The results were striking:

  • Superior Performance: On CIFAR-100 (ResNet32), the simple Post-scaling method achieved a final accuracy of 44.44%, outperforming established SOTA methods like iCaRL (42.95%) and EEIL (43.27%).
  • Efficiency: Unlike methods such as LSIL, which require additional training phases to learn bias parameters, post-scaling adds no overhead to the training process.
  • Baselines: The authors recommend that MDFCIL (Weight Aligning) and Post-scaling should serve as the primary baselines for future CIncL research due to their optimal balance of performance and simplicity.

Conclusion and Future Directions

The fact that basic logit adjustments from the CImbL field can match complex CIncL pipelines suggests that the "incremental" challenge may be largely a data quantity problem. However, the authors caution against total reductionism. While the final classifier's bias can be fixed with CImbL tricks, the underlying representation drift—how the feature space itself changes as new data arrives—remains a unique challenge for incremental learning. Future work should focus on maintaining robust representations rather than just patching the final decision layer.

Summary updated

Abstract

Catastrophic forgetting, the main challenge of Class Incremental Learning, is closely related to the classifier's bias due to imbalanced data, and most researchers resort to empirical techniques to remove the bias. Such anti-bias tricks share many ideas with the field of Class Imbalanced Learning, which encourages us to reflect on why these tricks work, and how we can design more principled solutions from a different perspective. In this paper, we comprehensively analyze the connections and seek possible collaborations between these two fields, i.e. Class Incremental Learning and Class Imbalanced Learning. Specifically, we first provide a panoramic view of recent bias correction tricks from the perspective of handling class imbalance. Then, we show that an adapted post-scaling technique which originates from Class Imbalanced Learning is on par with or even outperforms SOTA Class Incremental Learning method. Visualization via violin plots and polar charts further sheds light on how SOTA methods address the class imbalance problem from a more intuitive geometric perspective. These findings may encourage further infiltration between the two closely connected fields, but also raise concerns about whether it is correct that Class Incremental Learning degenerates into a class imbalance problem.
BibTeX
@inproceedings{he2021tale,
  author = {Chen He and Ruiping Wang and Xilin Chen},
  title = {{A Tale of Two CILs: The Connections Between Class Incremental Learning and Class Imbalanced Learning, and Beyond}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  url = {https://openreview.net/forum?id=og1ZdDheUYq},
}
Figure