TL;DR
This paper introduces a simple yet effective "Generalizable Two-Branch" (G2B) framework that enhances existing class-incremental learning (CIL) methods. By adding a lightweight "side branch" that modulates the main network's feature activations, G2B significantly reduces catastrophic forgetting and boosts performance across various CIL strategies and backbones.
What Problem Does This Paper Address?
Deep neural networks excel at learning from vast datasets, but they struggle to learn sequentially. When trained on new data, they tend to "catastrophically forget" what they learned previously. This is a major hurdle for real-world applications where models need to adapt and learn over time without access to all past data. Class-Incremental Learning (CIL) specifically addresses this by learning new classes sequentially, assuming limited or no access to data from previous classes. Existing CIL methods try to mitigate forgetting through various strategies like parameter regularization, data rehearsal, or dynamic model expansion, but there's still room for improvement.
Methods
The core innovation is the Generalizable Two-Branch (G2B) framework. It's designed to be a plug-in module that enhances any existing CIL method. The framework consists of:
- Main Branch: This is the existing CIL model (e.g., iCaRL, DER, DyTox) using either a CNN or Vision Transformer (ViT) backbone.
- Side Branch: A lightweight convolutional neural network (CNN) with a structure mirroring the main branch's blocks. Each side branch block is designed to be much simpler than its main branch counterpart.
The key interaction is that the output of each block in the main branch is modulated by the output of the corresponding side branch block. This modulation is achieved through a Hadamard product (element-wise multiplication), effectively acting as a "soft mask." An adapter function is used to ensure the spatial and channel dimensions of the side branch output match the main branch output before modulation. This modulation encourages sparser activations in the main branch, which the authors hypothesize helps preserve old knowledge. Crucially, G2B can be trained using the same vanilla optimization settings as the original CIL method, requiring no additional complex loss functions.
Datasets
The paper evaluates the proposed G2B framework on the following standard CIL benchmarks:
- CIFAR-100: A dataset of 100 object categories with 32x32 RGB images. Experiments were conducted with 10, 20, and 50 rounds of learning, using a fixed rehearsal memory of 2,000 exemplars.
- ImageNet-100: A subset of ImageNet containing 100 object categories.
- ImageNet-1000: The full ImageNet dataset with 1,000 object categories.
The CIL protocols from existing methods like DyTox and DER were followed for fair comparisons.
Key Insights and Ideas
- Simplicity is Key: The G2B framework is remarkably simple, acting as a lightweight add-on that doesn't require complex new loss functions or training procedures. This makes it highly practical.
- Modulation for Sparsity: The core idea of using a side branch to modulate the main branch's activations is novel. The resulting sparsity in feature representations is believed to be the mechanism that combats catastrophic forgetting.
- Generalizability: The framework's ability to enhance diverse CIL methods (like DER and DyTox) and work with different backbones (CNNs and ViTs) highlights its broad applicability.
- Consistent Improvement: G2B consistently improves performance across various datasets and CIL settings, often pushing state-of-the-art results even higher.
Key Results
- State-of-the-Art Performance: G2B consistently boosted the performance of existing SOTA CIL methods. For example, combining G2B with DER and DyTox led to significant improvements, often exceeding 1-2% accuracy gains on CIFAR-100.
- ImageNet Performance: Similar improvements were observed on ImageNet-100 and ImageNet-1000, demonstrating effectiveness on larger-scale datasets.
- Side Branch Impact: Experiments showed that even a simple side branch with two blocks provided noticeable improvements, and using more blocks (up to matching the main branch) yielded the best results.
- Model Size Insensitivity: G2B showed robust performance even with smaller model sizes, and its forgetting measure (F) was significantly lower than baseline methods.
- Efficiency: Compared to another two-branch approach (AANets), G2B achieved better performance while requiring less memory and computation.
Limitations and Future Work
- Limitations: The paper doesn't explicitly detail limitations beyond the inherent challenges of CIL. The performance gains, while consistent, are often in the range of a few percentage points, and further research might explore maximizing these gains.
- Future Work: The authors suggest that the G2B framework can be extended to other continual learning tasks, such as object detection and semantic segmentation. They also imply that further exploration of the side branch's architecture and its interaction with the main branch could yield even better results.
Summary updated