ScaIL: Classifier Weights Scaling for Class Incremental Learning

Eden Belouadah, Adrian PopescuWACV 2020

TL;DR

This paper introduces ScaIL, a novel method for class incremental learning that tackles the challenge of catastrophic forgetting in deep learning models. ScaIL achieves improved performance by scaling the weights of past classifiers to match those of new classes, and importantly, demonstrates that removing distillation losses can be beneficial, especially for large-scale datasets.

What Problem Does This Paper Address?

Artificial agents operating in dynamic environments often need to learn new information over time without forgetting what they've already learned. This is known as incremental learning (IL). A significant challenge in IL, especially for deep learning models, is "catastrophic forgetting," where learning new data causes the model to lose performance on previously learned data. This problem is exacerbated when agents have limited computational resources and memory to store past data. Existing IL methods often fall into two categories: those that allow the model's architecture to grow (increasing parameters) or those that keep the architecture fixed but use a memory buffer of past data. The latter approach, which this paper focuses on, often suffers from a bias towards new classes due to data imbalance, leading to decreased performance on older classes. This paper aims to mitigate this bias and improve overall IL performance within a fixed architecture and bounded memory constraint.

Methods

The core contribution of ScaIL is a novel approach to address the bias towards new classes in incremental learning. Instead of relying on data balancing techniques or complex distillation losses, ScaIL proposes Classifier Weights Scaling. The key idea is to leverage the classifiers learned during the initial state of each class (when all data for that class was available) and scale them to be more comparable to the classifiers of newly learned classes.

Here's a breakdown of the method:

  • Compact Memory for Initial Classifiers: ScaIL introduces a small, auxiliary memory (denoted as 'I') that stores the classifier weights from the initial states of past classes. This memory is significantly smaller than the exemplar memory ('B') used for storing past data samples.
  • Classifier Weight Reshaping: As the model progresses through incremental states, the initial classifiers stored in memory 'I' are "reshaped" to be compatible with the current model's feature space. This reshaping involves using aggregate statistics from both the current and initial states.
  • Scaling Mechanism: The scaling is achieved by computing a ratio between the mean activation of new classes and that of past classes in their initial state. This ratio is then applied to the weights of the past classifiers. The paper notes that only the top-k (experimentally set to 10) past classes are scaled, with others being set to zero.
  • Ablation of Distillation Loss: A significant departure from many existing IL methods is the removal of the widely used distillation loss. The authors hypothesize that distillation, while intended to prevent forgetting, can actually reinforce errors due to data imbalance in large-scale scenarios. ScaIL relies on fine-tuning with a standard classification loss.

Datasets

The paper evaluates ScaIL on four public datasets:

  • ILSVRC [27]: A subset of ImageNet, used for object recognition.
  • VGGFace2 [4]: A face recognition dataset.
  • Google Landmarks [21]: A landmark recognition dataset.
  • CIFAR-100 [14]: An object recognition dataset.

The authors use configurations that are comparable to previous work, varying the memory size (B) and the number of incremental states (Z) to assess the robustness of their method.

Key Insights and Ideas

  • Distillation Can Hurt Performance: The most surprising finding is that for large-scale datasets, the widely used distillation loss can actually degrade performance in incremental learning. This suggests that distillation might amplify existing biases rather than mitigating forgetting.
  • Initial Classifiers Hold Valuable Information: ScaIL demonstrates that the initial classifiers, learned with complete data, contain valuable information that can be effectively repurposed for later incremental states through careful scaling.
  • Simplicity Can Be Powerful: By removing distillation and introducing a simple weight scaling mechanism, ScaIL achieves state-of-the-art results, highlighting that complex regularization techniques are not always necessary.
  • Stability Across Configurations: ScaIL shows remarkable stability across different memory sizes and numbers of incremental states, making it a robust solution for practical IL scenarios.

Key Results

  • ScaIL consistently outperforms strong baselines, including iCaRL and BiC, across the tested datasets.
  • On larger datasets like ILSVRC, VGGFace2, and Landmarks, ScaIL achieves a significantly better GIL (Global performance score) compared to iCaRL, indicating a smaller performance gap with full learning.
  • The paper reports that ScaIL improves FT (vanilla fine-tuning) performance by 2.69 GIL points.
  • The removal of distillation (ScaIL vs. FT) leads to performance improvements, particularly on larger datasets, with FT outperforming iCaRL by a substantial margin.
  • ScaIL is shown to be the most stable method across varying memory (B) and state (Z) configurations.

Limitations and Future Work

  • Top-k Scaling: The decision to scale only the top-k past class classifiers is an experimental choice. Further investigation into optimal k values or alternative selection strategies could be beneficial.
  • Real-world Scenarios: The paper acknowledges that the current work focuses on fully labeled incremental streams. Future work aims to address more realistic scenarios where streamed data might be partially or completely unlabeled.
  • Vanilla FT Improvement: While ScaIL is effective, the paper suggests that further improvements to vanilla fine-tuning itself, while keeping model complexity and memory bounded, could lead to even better performance.
  • Data Imbalance Handling: While ScaIL addresses bias through weight scaling, exploring new ways to handle data imbalance more directly remains an open area.

Summary updated

BibTeX
@inproceedings{belouadah2020scail,
  author = {Eden Belouadah and Adrian Popescu},
  title = {{ScaIL: Classifier Weights Scaling for Class Incremental Learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure