Abstract
Continual Learning (CL) aims to enable AI models to learn tasks sequentially without catastrophic forgetting. However, real-world data is often imbalanced and arrives in "blurry" streams. This paper presents AIR (Analytic Imbalance Rectifier), an exemplar-free algorithm that addresses these challenges using a closed-form analytic solution. By introducing an Analytic Re-weighting Module (ARM), AIR ensures that every class contributes equally to the model's optimization, effectively neutralizing the bias toward frequent "head" classes. Experimental results demonstrate that AIR significantly outperforms state-of-the-art methods in long-tailed and generalized incremental learning scenarios while maintaining memory efficiency and privacy.
Introduction
Standard Class-Incremental Learning (CIL) methods typically assume balanced data distributions. In practice, data is often long-tailed—where a few classes have many samples and many classes have very few—or arrives in "blurry" streams where task boundaries are indistinct. These conditions exacerbate two primary issues: Catastrophic Forgetting, where new knowledge overwrites the old, and Data Imbalance, where models become biased toward frequent classes.
Existing solutions often rely on Exemplar Replay, storing a subset of old data to mitigate forgetting. However, this approach consumes memory and poses privacy risks. Gradient-based methods also suffer from "task-recency bias," favoring the most recently seen data. AIR overcomes these limitations by utilizing an analytic approach that avoids gradient descent entirely for the classifier layer.
Methodology: The Analytic Imbalance Rectifier
AIR builds upon Analytic Continual Learning (ACL). Instead of iterative backpropagation, ACL uses a frozen, pre-trained backbone to extract features and solves for optimal classifier weights using ridge regression.
The core innovation is the Analytic Re-weighting Module (ARM). Standard ACL treats every sample equally, meaning a class with 100 samples has 100 times the influence of a class with one sample. AIR introduces a scalar weighting factor ($\pi_y$) for each class, set to the reciprocal of the class's sample count. This mathematical adjustment forces the optimization process to treat every class with equal importance.
Because it relies on a closed-form solution, AIR can update its knowledge incrementally by storing and updating auto-correlation and cross-correlation matrices. This "exemplar-free" nature makes it highly efficient and privacy-compliant.
Experimental Setup and Results
The authors evaluated AIR on several benchmarks designed to simulate real-world imbalance, including CIFAR-100 (LT) and ImageNet-R (LT) with imbalance ratios up to 1:500, and Generalized CIL (Si-blurry) settings where old and new classes appear simultaneously.
AIR consistently outperformed state-of-the-art (SOTA) methods. On CIFAR-100 (LT), AIR improved last-phase accuracy ($A_{last}$) by over 7% compared to the next best method. Furthermore, AIR demonstrated remarkable weight invariance: unlike gradient-based methods, it achieves nearly identical accuracy regardless of whether it sees "rich" or "scarce" classes first.
Key Insights: Rectifying the "Head" Bias
Standard ACL methods tend to produce larger weight norms for head classes, leading to biased predictions. AIR successfully rectifies this distribution, ensuring that the model does not ignore rare but critical categories.
As shown in the confusion matrices, AIR maintains high accuracy across the entire class spectrum, whereas standard ACIL shows significant confusion and bias toward earlier or more frequent classes. By mathematically finding the global optimum for all data seen so far, AIR bypasses the recency bias inherent in traditional training.
Conclusion
AIR provides a mathematically elegant and efficient solution for imbalanced continual learning. By replacing gradient descent with an analytic re-weighted solution, it eliminates task-recency bias and handles long-tailed distributions without needing to store raw data. While memory requirements scale with class count and feature dimensions, AIR represents a significant step toward robust, privacy-preserving AI that can learn effectively from messy, real-world data streams.
Summary updated