DILF-EN framework for Class-Incremental Learning

Mohammed Asad Karim, Indu Joshi, Pratik Mazumder, Pravendra SingharXiv 2021 · arXiv

Introduction

Deep learning models typically operate under the assumption of a static dataset. However, real-world applications often require models to learn from a continuous stream of data where new classes appear over time without full access to previous training samples. This paradigm, known as Class-Incremental Learning (CIL), is plagued by catastrophic forgetting: as a model adapts to new information, it tends to overwrite the weights necessary for recognizing older classes, leading to a sharp decline in performance on previous tasks.

To address this, the DILF-EN framework introduces a novel approach that leverages image orientations as a stable "anchor" to preserve knowledge across learning phases.

The DILF-EN Framework

The framework consists of two synergistic components: Dual-Incremental Learning (DILF) and Data-Ensemble (EN).

Dual-Incremental Learning (DILF)

Standard CIL models focus solely on predicting image classes. DILF augments this by adding a secondary objective: orientation classification. During training, images are rotated (e.g., 0°, 90°, 180°, 270°) and assigned two labels: their semantic class and their rotation class.

While the set of image classes grows with each phase, the orientation classes remain constant. This consistency forces the backbone feature extractor to maintain stable, generalizable representations that are less susceptible to the weight drift caused by learning new, volatile image classes.

Data-Ensemble (EN)

At inference time, the model utilizes a form of Test-Time Augmentation (TTA) optimized for the DILF regime. Instead of a single pass, the model processes multiple orientations of a test image, generates class probabilities for each, and averages them to produce a final prediction.

Overview of the DILF-EN framework showing the dual-head training process and the ensemble testing procedure.

Key Insights

  • Orientation-Dependent Forgetting: A core discovery of this research is that a model may "forget" a class in its original orientation while still correctly identifying it when rotated. This suggests that class knowledge is often inconsistent rather than entirely lost.
  • The "Anchor" Effect: Orientation serves as a fixed target. By training on a task that never changes, the model anchors its weights, preventing them from drifting too far during incremental updates.
  • Mandatory Synergy: The authors emphasize that Data-Ensemble (EN) is ineffective without DILF training. Standard models, not trained on rotated inputs, often see a performance drop when subjected to ensemble testing across orientations.

Experimental Results

The framework was evaluated on standard benchmarks, including CIFAR-100 and ImageNet-100, across various phase counts ($P=5, 10, 25$). DILF-EN was integrated into several state-of-the-art (SOTA) baselines, including AANets, PODNet, and LUCIR.

Performance comparison showing significant accuracy gains when DILF-EN is applied to various SOTA incremental learning methods.

As shown in the results, DILF-EN consistently improves performance across all settings. For instance, on CIFAR-100, it boosted the performance of AANets by absolute margins ranging from 3.30% to 4.28%. Visual analysis via Grad-CAM further revealed that DILF-EN helps the model maintain focus on discriminative object features rather than drifting toward background noise after multiple learning phases.

Limitations and Future Work

While effective, the framework introduces computational overhead. Training requires larger effective batch sizes to accommodate rotated images, and inference time increases due to the multi-pass ensemble process. Future research may explore other self-supervised tasks beyond rotation to provide even more robust anchors against catastrophic forgetting.

Summary updated

Abstract

Deep learning models suffer from catastrophic forgetting of the classes in the older phases as they get trained on the classes introduced in the new phase in the class-incremental learning setting. In this work, we show that the effect of catastrophic forgetting on the model prediction varies with the change in orientation of the same image, which is a novel finding. Based on this, we propose a novel data-ensemble approach that combines the predictions for the different orientations of the image to help the model retain further information regarding the previously seen classes and thereby reduce the effect of forgetting on the model predictions. However, we cannot directly use the data-ensemble approach if the model is trained using traditional techniques. Therefore, we also propose a novel dual-incremental learning framework that involves jointly training the network with two incremental learning objectives, i.e., the class-incremental learning objective and our proposed data-incremental learning objective. In the dual-incremental learning framework, each image belongs to two classes, i.e., the image class (for class-incremental learning) and the orientation class (for data-incremental learning). In class-incremental learning, each new phase introduces a new set of classes, and the model cannot access the complete training data from the older phases. In our proposed data-incremental learning, the orientation classes remain the same across all the phases, and the data introduced by the new phase in class-incremental learning acts as new training data for these orientation classes. We empirically demonstrate that the dual-incremental learning framework is vital to the data-ensemble approach. We apply our proposed approach to state-of-the-art class-incremental learning methods and empirically show that our framework significantly improves the performance of these methods.
BibTeX
@misc{karim2021dilf,
  author = {Mohammed Asad Karim and Indu Joshi and Pratik Mazumder and Pravendra Singh},
  title = {{DILF-EN framework for Class-Incremental Learning}},
  year = {2021},
  eprint = {2112.12385},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2112.12385},
}
Figure