BEEF: Bi-Compatible Class-Incremental Learning via Energy-Based Expansion and Fusion

Fu-Yun Wang, Da-Wei Zhou, Liu Liu, Han-Jia Ye, Yatao Bian, De-Chuan Zhan, Peilin ZhaoICLR 2023 · OpenReview

TL;DR

BEEF (Bi-Compatible Energy-Based Expansion and Fusion) is a novel framework for class-incremental learning that uses energy-based theory to achieve "bi-directional compatibility." By training independent modules with specialized prototypes for past and future tasks, it effectively eliminates catastrophic forgetting and remains remarkably robust even when the "memory" of old data is messy and imbalanced.

What Problem Does This Paper Address?

In the world of AI, Class-Incremental Learning (CIL) is a major hurdle. Ideally, a model should learn new categories (like "panda" and "koala") over time without forgetting what it already knows (like "cat" and "dog"). However, standard neural networks suffer from catastrophic forgetting: when fine-tuned on new data, they abruptly lose their ability to recognize old classes.

Existing solutions generally fall into two camps:

  1. Regularization-based: They try to "freeze" important parameters, which often limits the model's ability to learn new things (the stability-plasticity dilemma).
  2. Dynamic-structure-based: They add new "modules" (sub-networks) for every new task. While powerful, these modules are usually trained in a coupled way, leading to conflicts. For example, an old module might mistakenly claim a new image belongs to an old class because it was never taught that "new things" exist.

BEEF addresses these conflicts by ensuring bi-directional compatibility: making sure new modules don't hurt old ones (backward compatibility) and that old modules don't overconfidently "steal" predictions from future classes (forward compatibility).

Methods

BEEF introduces a two-stage training approach: Expansion and Fusion.

The BEEF framework illustrating Model Expansion with independent modules and prototypes, followed by Model Fusion.

1. Model Expansion (The "Energy" Part)

Instead of training everything together, BEEF trains a new module for the current task independently. To handle the "open world" (classes it hasn't seen yet or has already seen), it introduces two virtual prototypes:

  • Backward Prototype ($p_b$): Acts as a "cluster" for all old classes. This helps the new module learn a clear boundary between the "current task" and "everything from the past."
  • Forward Prototype ($p_f$): Acts as a placeholder for future, unseen classes.

The authors treat the classifier as an Energy-Based Model (EBM). During training, they use a technique called Stochastic Gradient Langevin Dynamics (SGLD) to generate "fake" samples from the current model's distribution. They then train the model to assign high energy (low probability) to these generated samples and low energy to real samples. This makes the model "aware" of its own uncertainty, preventing it from being overconfident on data it doesn't recognize.

2. Model Fusion

Once the new module is trained, it needs to be integrated with the old ones. BEEF uses a small "exemplar set" (a tiny memory of old images) to learn a simple affine transformation (scaling and shifting). This balances the outputs of the different modules, mitigating "task bias"—the tendency of the model to favor the most recently learned classes.

Datasets

The authors evaluated BEEF on three standard computer vision benchmarks:

  • CIFAR-100: 100 classes of small images.
  • ImageNet-100: A subset of 100 high-resolution classes.
  • ImageNet-1000: The full-scale industry standard with 1,000 classes.

Crucially, they also tested a "Challenging CIL" setting where the exemplar set is randomly selected and highly imbalanced, mimicking real-world scenarios where data might be missing or poorly curated.

Key Insights and Ideas

  • Decoupling is Possible: You don't need to train all modules simultaneously to avoid forgetting. If you provide the right "placeholders" ($p_b$ and $p_f$), modules can be trained in isolation and fused later with minimal cost.
  • Modeling $P(x)$ Matters: Most classifiers only care about $P(y|x)$ (the probability of a label given an image). By using EBM theory to model $P(x)$ (the distribution of the images themselves), the model learns to recognize when an input is "out-of-distribution" for its specific task.
  • Robustness to "Messy" Memory: Most CIL methods break down if their small memory of old images isn't perfectly balanced. BEEF’s use of a single backward prototype for all old classes makes it much less sensitive to whether it has 5 or 500 examples of a "cat."

Key Results

Performance comparison on ImageNet-100 and ImageNet-1000 benchmarks showing BEEF achieving state-of-the-art results.
  • State-of-the-Art (SOTA): BEEF consistently outperformed top-tier methods like DER and FOSTER across all benchmarks. On CIFAR-100 (B50 protocol), it improved average accuracy by up to 5.45% over previous methods.
  • Resilience: In the imbalanced exemplar setting, while other methods saw their performance "fail dramatically," BEEF maintained high accuracy, often outperforming competitors by more than 10%.
  • Efficiency: Because modules are trained independently, the training time for BEEF is lower than many other dynamic-structure methods that require forward passes through all previous modules.

Limitations and Future Work

  • Parameter Growth: Like other dynamic-structure methods, BEEF adds new parameters for every task. While the authors propose a "BEEF-Compress" version to squash these into a single backbone, this compression step adds its own complexity.
  • SGLD Overhead: Generating samples from the energy manifold via SGLD is more computationally expensive than standard training, though the authors implemented several strategies (like mixup and rotation) to speed this up.
  • Future Direction: The authors suggest exploring how to better select the number of forward prototypes in "Stable-BEEF" to further improve performance on very long-term incremental tasks.

Summary updated

Abstract

Neural networks suffer from catastrophic forgetting when sequentially learning tasks phase-by-phase, making them inapplicable in dynamically updated systems. Class-incremental learning (CIL) aims to enable neural networks to learn different categories at multi-stages. Recently, dynamic-structure-based CIL methods achieve remarkable performance. However, these methods train all modules in a coupled manner and do not consider possible conflicts among modules, resulting in spoilage of eventual predictions. In this work, we propose a unifying energy-based theory and framework called Bi-Compatible Energy-Based Expansion and Fusion (BEEF) to analyze and achieve the goal of CIL. We demonstrate the possibility of training independent modules in a decoupled manner while achieving bi-directional compatibility among modules through two additionally allocated prototypes, and then integrating them into a unifying classifier with minimal cost. Furthermore, BEEF extends the exemplar-set to a more challenging setting, where exemplars are randomly selected and imbalanced, and maintains its performance when prior methods fail dramatically. Extensive experiments on three widely used benchmarks: CIFAR-100, ImageNet-100, and ImageNet-1000 demonstrate that BEEF achieves state-of-the-art performance in both the ordinary and challenging CIL settings. The Code is available at https://github.com/G-U-N/ICLR23-BEEF.
BibTeX
@inproceedings{wang2023beef,
  author = {Fu-Yun Wang and Da-Wei Zhou and Liu Liu and Han-Jia Ye and Yatao Bian and De-Chuan Zhan and Peilin Zhao},
  title = {{BEEF: Bi-Compatible Class-Incremental Learning via Energy-Based Expansion and Fusion}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=iP77_axu0h3},
}
Figure