Elastic Feature Consolidation for Cold Start Exemplar-Free Incremental Learning

Simone Magistri, Tomaso Trinci, Albin Soutif-Cormerais, Joost van de Weijer, Andrew D. Bagdanov

Class-Incremental Learning (CIL) is the holy grail of flexible AI: the ability for a model to learn new tasks sequentially without forgetting what it already knows. However, most current methods rely on a "Warm Start"—an initial large task that provides a solid, stable foundation. But what happens when your AI starts with only a handful of examples?

In their ICLR 2024 paper, "Elastic Feature Consolidation for Cold Start Exemplar-Free Incremental Learning," Simone Magistri and colleagues tackle the "Cold Start" problem head-on, offering a way to maintain high model plasticity without sacrificing stability.

TL;DR

Elastic Feature Consolidation (EFC) enables effective exemplar-free incremental learning even when the initial training task is small (the "Cold Start" scenario). By using a novel Empirical Feature Matrix to selectively regularize feature drift, EFC allows the model's backbone to adapt to new data while protecting the representations critical for previous tasks.

What Problem Does This Paper Address?

Most Exemplar-Free Class Incremental Learning (EFCIL) methods—which don't store old data due to privacy or storage constraints—assume a "Warm Start." They train on a massive first task (e.g., 50% of a dataset) to build a high-quality, often frozen, feature extractor (backbone).

In a Cold Start scenario, the first task is small. The backbone learned is insufficient and must adapt significantly as new tasks arrive. This creates a dilemma:

  1. High Plasticity: The backbone changes to learn new tasks, but this causes "feature drift," leading to catastrophic forgetting of the old, small tasks.
  2. High Stability: Traditional regularization (like feature distillation) keeps the backbone stable but prevents it from learning the new features necessary for subsequent tasks.

EFC aims to break this deadlock by being "elastic"—rigid where it matters for old tasks, and flexible everywhere else.

Methods

The authors propose a framework consisting of three main components, illustrated in the architecture overview below:

Figure 1: Elastic Feature Consolidation. (a) Architecture overview; (b) The Empirical Feature Matrix (EFM) measures how outputs vary with features and identifies important directions to mitigate forget- ting (Section 3.3); (c) The EFM induces a pseudo-metric in feature space used to estimate prototype drift (Section 4.3); and (d) The Asymmetric Prototype Replay loss adapts previous task classifiers to the changing backbone by balancing new-task data and Gaussian prototypes (Section 4.2).
  1. Empirical Feature Matrix (EFM): Inspired by Elastic Weight Consolidation (EWC), which regularizes weights, EFC regularizes features. The EFM identifies specific directions in the feature space that are most relevant to previous tasks. Instead of penalizing all changes (isotropic regularization), EFC only penalizes drift in these "relevant" directions, leaving other directions "free" for the backbone to adapt to new tasks.
  2. Asymmetric Prototype Replay (PR-ACE): Since no old images are stored, EFCIL methods use "prototypes" (class means and covariances) to represent old classes. Standard methods treat these prototypes and new data symmetrically. EFC introduces an asymmetric cross-entropy loss that better balances the learning of new task-specific features with the calibration needed to keep old class boundaries intact.
  3. Prototype Drift Compensation: As the backbone evolves, the stored prototypes of old classes become inaccurate. EFC uses the EFM to intelligently update these prototypes based on how the feature space is shifting, ensuring they remain representative even as the model changes.

Datasets

The method was evaluated on four standard benchmarks:

  • CIFAR-100
  • Tiny-ImageNet
  • ImageNet-Subset (100 classes)
  • ImageNet-1K (The full 1,000-class challenge)

Crucially, the authors tested these in both Warm Start (large first task) and the much harder Cold Start (all tasks are the same size) scenarios.

Key Insights and Ideas

  • Feature Space vs. Weight Space: While EWC is elegant, calculating the Fisher Information Matrix for millions of parameters is computationally brutal. EFM operates in the feature space (usually much lower dimensional, e.g., 512), making it tractable and efficient.
  • The Elbow of the Spectrum: The authors found that the rank of the EFM corresponds closely to the number of classes seen so far. This means the model only needs to "protect" a small number of dimensions in the feature space, leaving the vast majority of the space free for new learning.
  • Plasticity is Mandatory for Cold Start: The paper demonstrates that methods relying on feature distillation or frozen backbones fail miserably in Cold Start scenarios because they cannot adapt to the new information presented in later tasks.

Key Results

EFC significantly outperforms state-of-the-art EFCIL methods (like PASS, SSRE, and FeTrIL) across the board, with the most dramatic gains in Cold Start scenarios.

  • ImageNet-Subset (Cold Start, 10-step): EFC achieved 47.38% per-step accuracy, compared to 36.17% for the next best method (FeTrIL).
  • ImageNet-1K (Cold Start, 10-step): EFC maintained its lead on the full-scale dataset, outperforming FeTrIL by over 8% in average incremental accuracy.
  • Ablation Success: The authors showed that the combination of EFM and the asymmetric PR-ACE loss is critical. Using EFM with a standard symmetric loss actually hurts performance, proving that the two components are designed to work in tandem.

Limitations and Future Work

  • Storage Scaling: EFC requires storing class covariance matrices, which grows linearly with the number of classes. While the authors suggest low-rank approximations to mitigate this, it remains a factor for extremely long task sequences.
  • Prototype Drift: While EFC compensates for drift, estimating the drift of class means over very long sequences without any original data remains an open research question.

For those interested in the implementation, the code is available at https://github.com/simomagi/elastic_feature_consolidation.

Summary updated

Abstract

Exemplar-Free Class Incremental Learning (EFCIL) aims to learn from a sequence of tasks without having access to previous task data. In this paper, we consider the challenging Cold Start scenario in which insufficient data is available in the first task to learn a high-quality backbone. This is especially challenging for EFCIL since it requires high plasticity, which results in feature drift which is difficult to compensate for in the exemplar-free setting. To address this problem, we propose a simple and effective approach that consolidates feature representations by regularizing drift in directions highly relevant to previous tasks and employs prototypes to reduce task-recency bias. Our method, called Elastic Feature Consolidation (EFC), exploits a tractable second-order approximation of feature drift based on an Empirical Feature Matrix (EFM). The EFM induces a pseudo-metric in feature space which we use to regularize feature drift in important directions and to update Gaussian prototypes used in a novel asymmetric cross entropy loss which effectively balances prototype rehearsal with data from new tasks. Experimental results on CIFAR-100, Tiny-ImageNet, ImageNet-Subset and ImageNet-1K demonstrate that Elastic Feature Consolidation is better able to learn new tasks by maintaining model plasticity and significantly outperform the state-of-the-art.
BibTeX
@inproceedings{magistri2024elastic,
  author = {Simone Magistri and Tomaso Trinci and Albin Soutif-Cormerais and Joost van de Weijer and Andrew D. Bagdanov},
  title = {{Elastic Feature Consolidation for Cold Start Exemplar-Free Incremental Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2024},
  url = {https://openreview.net/forum?id=7D9X2cFnt1},
}
Figure