Abstract: Catastrophic forgetting, the main challenge of Class Incremental Learning (CIncL), is closely related to the classifier's bias due to imbalanced data. Most researchers resort to empirical techniques to remove this bias, yet these "anti-bias" tricks share many ideas with the field of Class Imbalanced Learning (CImbL). This paper comprehensively analyzes the connections between these two fields. We provide a panoramic view of recent bias correction tricks from the perspective of handling class imbalance and demonstrate that an adapted post-scaling technique from CImbL is on par with or even outperforms state-of-the-art CIncL methods. Visualization via violin plots and polar charts further sheds light on how these methods address class imbalance from a geometric perspective. Our findings raise the question of whether CIncL is essentially degenerating into a class imbalance problem.
Introduction: Are We Reinventing the Wheel?
In Class Incremental Learning (CIncL), a model must learn new categories in stages without forgetting old ones. Due to memory constraints, we can only store a small fraction of old data (exemplars), leading to a training set that is massively biased toward the newest classes. This imbalance is a primary driver of catastrophic forgetting, where the model’s accuracy on previously learned classes plummets.
While the CIncL community has developed numerous "anti-bias" tricks to mitigate this, these methods are often empirical and disconnected from the established field of Class Imbalanced Learning (CImbL). This research seeks to bridge that gap, asking if CIncL is simply a specific, severe instance of the class imbalance problem.
Methodology: Mapping the Connection
The authors first establish a systematic mapping between popular CIncL methods and classic CImbL strategies. As shown in the table below, many "innovations" in incremental learning have direct counterparts in the imbalance literature, such as under-sampling, two-phase training, and probability calibration.
The Post-Scaling Technique
Building on this connection, the authors introduce a "post-scaling" layer derived from Bayesian theory. Instead of complex training routines, they apply a logit-level bias term at test time:
$$\log\left(\frac{p_{ts}(c)}{p_{tr}(c)}\right)$$
This adjusts the model's predictions based on the ratio of samples seen during training ($p_{tr}$) versus the expected distribution at test time ($p_{ts}$). This simple adjustment requires virtually zero training overhead yet addresses the core imbalance issue directly.
Geometric Insights into Bias
To understand why these techniques work, the paper employs geometric visualizations. By using polar charts to visualize the angular distribution of features, the researchers demonstrate that classifier bias is not just a numerical error but a physical shift in decision boundaries.
In an imbalanced scenario, the decision boundary naturally "shrinks" around the minority (old) classes. Techniques like post-scaling and weight aligning effectively "push" or "rotate" these boundaries to restore a fair volume for the minority classes in the feature space.
Experimental Results
The study evaluated these methods on standard benchmarks including CIFAR-100 and a 100-class subset of ImageNet. The results were striking:
- Superior Performance: On CIFAR-100 (ResNet32), the simple Post-scaling method achieved a final accuracy of 44.44%, outperforming established SOTA methods like iCaRL (42.95%) and EEIL (43.27%).
- Efficiency: Unlike methods such as LSIL, which require additional training phases to learn bias parameters, post-scaling adds no overhead to the training process.
- Baselines: The authors recommend that MDFCIL (Weight Aligning) and Post-scaling should serve as the primary baselines for future CIncL research due to their optimal balance of performance and simplicity.
Conclusion and Future Directions
The fact that basic logit adjustments from the CImbL field can match complex CIncL pipelines suggests that the "incremental" challenge may be largely a data quantity problem. However, the authors caution against total reductionism. While the final classifier's bias can be fixed with CImbL tricks, the underlying representation drift—how the feature space itself changes as new data arrives—remains a unique challenge for incremental learning. Future work should focus on maintaining robust representations rather than just patching the final decision layer.
Summary updated