Split-and-Bridge: Adaptable Class Incremental Learning within a Single Neural Network

Jong-Yeong Kim, Dong-Wan ChoiAAAI 2021 · DOI

Introduction

In the pursuit of artificial general intelligence, Class Incremental Learning (CIL) remains a formidable challenge. Unlike traditional machine learning, which assumes access to all data simultaneously, CIL requires models to learn new classes sequentially without forgetting previously acquired knowledge. This must be achieved while maintaining a single, unified classifier. Researchers from Inha University have proposed a novel architectural strategy called Split-and-Bridge (S&B) to resolve the inherent conflict between preserving old knowledge and acquiring new information.

The Challenge: Stability vs. Plasticity

The primary mechanism used to prevent "catastrophic forgetting" in CIL is Knowledge Distillation (KD). KD encourages a model learning a new task to produce outputs similar to its previous state. However, the authors identify a critical "KD Dominance" issue: because KD labels contain more information than simple one-hot class labels, the KD loss often overpowers the learning signal for new classes.

The three types of knowledge in CIL: intra-old, intra-new, and cross-task knowledge.

As illustrated above, CIL requires mastering three distinct types of knowledge:

  1. Intra-old knowledge: Discriminating between previously learned classes.
  2. Intra-new knowledge: Discriminating between newly introduced classes.
  3. Cross-task knowledge: Discriminating between old and new classes.

Standard KD-based methods often become too "stiff," sacrificing intra-new and cross-task performance to maintain intra-old stability.

The Split-and-Bridge Method

To address this, the S&B approach introduces a two-phase learning process that physically partitions the network to reduce gradient interference.

Architectural comparison between standard KD-based CIL and the proposed Split-and-Bridge method.

1. The Split Phase

In this phase, the network's upper layers are partially split into two disjoint branches—one dedicated to old classes and one to new classes. They continue to share lower-level feature extraction layers. To ensure a clean separation, the authors employ weight sparsification ($l_2$ regularization) to "hollow out" connections between the intended branches. Furthermore, they introduce Localized Cross Entropy (LCE), which restricts the new branch's learning to the probability space of the new task, preventing interference from the old task's output space.

2. The Bridge Phase

Once the branches have independently mastered their respective tasks, they are re-connected. Crucially, the new "bridge" weights are initialized to zero. This prevents random noise from destroying the finely-tuned features developed during the split phase. The model then undergoes a final training pass to integrate the branches and learn the necessary cross-task knowledge to unify the classifier.

Experimental Results

The S&B method was evaluated on standard benchmarks, including CIFAR-100 and Tiny-ImageNet, using a ResNet-18 backbone.

  • Superior Performance: On CIFAR-100 (10 tasks), S&B achieved 66.97% average accuracy, significantly outperforming standard KD-based methods like Weight Aligning (WA) at 63.72%.
  • Balanced Plasticity: Unlike competitors that sacrifice new class accuracy for stability, S&B maintained high performance across both. On Tiny-ImageNet, S&B's intra-new accuracy was consistently 5-10% higher than state-of-the-art baselines.
  • Incremental Robustness: The method proved particularly robust in scenarios with very small task increments (e.g., 20 tasks of 5 classes), where other methods typically collapse.

Conclusion

Split-and-Bridge offers a structural solution to the stability-plasticity dilemma in CIL. By isolating learning through architectural splitting and carefully reintegrating knowledge via zero-initialized bridging, the model avoids the stifling effects of dominant KD signals. Future work will explore applying this logic to more complex domains like object detection and sequence generation.

Summary updated

BibTeX
@inproceedings{kim2021split,
  author = {Jong-Yeong Kim and Dong-Wan Choi},
  title = {{Split-and-Bridge: Adaptable Class Incremental Learning within a Single Neural Network}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i9.16991},
}
Figure