TL;DR
This paper introduces TACLE, a novel framework for "exemplar-free semi-supervised class incremental learning" (EFSS-CIL). TACLE tackles the challenging problem of learning new classes over time without storing old data, leveraging pre-trained models and novel techniques to effectively utilize unlabeled data for improved performance.
What Problem Does This Paper Address?
The paper addresses the critical challenge of Class Incremental Learning (CIL), where a model must learn new classes sequentially without forgetting previously learned ones. A more practical and challenging variant is Semi-Supervised Class Incremental Learning (SS-CIL), where only a few labeled samples are available for each new task, and the majority of the data is unlabeled. The most demanding scenario, and the focus of this paper, is Exemplar-Free SS-CIL (EFSS-CIL), which prohibits storing any data from previous tasks, making it ideal for privacy-sensitive applications or scenarios with limited storage. Traditional SS-CIL methods often rely on storing exemplars, which is not feasible in EFSS-CIL. Existing exemplar-free methods often struggle with effectively utilizing the abundant unlabeled data, especially as the number of classes grows, leading to performance degradation.
Methods
TACLE is a two-stage framework designed for EFSS-CIL, building upon pre-trained models.
- Stage 1: Feature Representation Learning:
- Task-wise Adaptive Threshold: Inspired by semi-supervised learning techniques like FixMatch, TACLE uses a confidence threshold to select unlabeled data for training. However, instead of a fixed threshold, TACLE employs a task-wise adaptive threshold. This threshold dynamically adjusts based on the task, accounting for the increasing class confusion and decreasing confidence scores of unlabeled data as more tasks are learned. This ensures that more unlabeled data is effectively utilized as learning progresses.
- Class-Aware Weighted Cross-Entropy Loss: To address class imbalance within a task (where some classes might have more confident pseudo-labels than others), TACLE introduces a class-aware weighted cross-entropy loss. This loss assigns higher weights to under-represented classes, improving their performance.
- Stage 2: Classifier Alignment:
- Exploiting Unlabeled Data for Classifier Alignment: This stage leverages the learned features and, crucially, the confident unlabeled samples to align the classifiers. By incorporating these unlabeled samples, TACLE can better estimate class statistics (mean and variance) and align classifiers more robustly, especially in low-data regimes. This is a key component for achieving optimal performance in EFSS-CIL.
Datasets
The proposed TACLE framework was evaluated on three widely used SS-CIL datasets:
- CIFAR10: A dataset of 32x32 images with 10 classes. For SS-CIL, it was structured into 5 tasks, with 2 new classes per task.
- CIFAR100: A dataset of 32x32 images with 100 classes. It was structured into 10 tasks, with 10 new classes per task. Images were resized to 224x224 for compatibility with pre-trained models.
- ImageNet-Subset100: A subset of ImageNet-1k with 100 classes. Images were resized to 256x256 and cropped to 224x224. This dataset was structured into 20 tasks, with 5 new classes per task.
The paper also explored different supervision levels for labeled data (0.8%, 5%, and 25% for CIFAR datasets, and 1%, 5%, and 25% for ImageNet-Subset100).
Key Insights and Ideas
- The power of adaptive thresholds: A fixed confidence threshold for unlabeled data is insufficient in incremental learning. As the model learns more tasks and faces more classes, its confidence in unlabeled data naturally decreases. An adaptive threshold that adjusts per task is crucial for maximizing the utility of unlabeled data.
- Class imbalance is a persistent problem: Even with unlabeled data, within-task class imbalance can skew learning. Explicitly weighting under-represented classes in the loss function is a simple yet effective way to mitigate this.
- Unlabeled data is valuable for classifier alignment: Traditional classifier alignment relies on limited labeled data. Incorporating confident unlabeled samples in this stage significantly boosts performance, especially when labeled data is scarce.
- Pre-trained models are essential: Leveraging pre-trained models provides a strong foundation, enabling TACLE to achieve impressive results in the challenging EFSS-CIL setting.
Key Results
TACLE demonstrates significant improvements over baseline methods, particularly SLCA, across various datasets and supervision levels.
- On CIFAR10, with only 0.8% labeled data, TACLE achieved a substantial 7% improvement over SLCA using a pre-trained model.
- On CIFAR100, TACLE showed improvements of up to 13.08% over SLCA with 0.8% labeled data.
- In extreme scenarios:
- One-shot EFSS-CIL (one labeled example per class) on CIFAR100 showed improvements of 25.77% (ImageNet pre-trained) and 7.67% (MoCo v3 pre-trained) over SLCA.
- Imbalanced SS-CIL settings also highlighted TACLE's effectiveness in handling skewed unlabeled data distributions.
- TACLE consistently outperformed baselines, especially in low-data regimes, underscoring the effectiveness of its proposed components.
Limitations and Future Work
- Assumption of current task data: TACLE, like many SS-CIL methods, assumes that unlabeled data exclusively belongs to the current task. Real-world scenarios might involve mixed data sources, including samples from previous tasks or outliers.
- Future directions: The authors suggest exploring more realistic settings where unlabeled data might be mixed or contain outliers. Further investigation into the impact of hyper-parameters on the adaptive threshold was also mentioned, with detailed results provided in supplementary material.
Summary updated