Co-Transport for Class-Incremental Learning

Da-Wei Zhou, Han-Jia Ye, De-Chuan Zhan

Abstract

Class-Incremental Learning (CIL) is essential for deploying deep learning models in dynamic, open-world environments where new categories emerge over time. However, CIL is hindered by catastrophic forgetting and the suboptimal initialization of new class classifiers. This paper introduces COIL (CO-transport for class Incremental Learning), a novel framework that utilizes Optimal Transport (OT) to bridge the semantic gap between learning stages. By establishing a bi-directional knowledge flow—Prospective Transport for "warm-starting" new classes and Retrospective Transport for reinforcing old ones—COIL significantly enhances model stability and adaptation speed. Experimental results on benchmarks like CIFAR-100 and ImageNet demonstrate that COIL consistently outperforms existing state-of-the-art methods.

Introduction

Standard deep learning models operate under a "closed-world" assumption, where the set of classes is fixed during training. In contrast, real-world applications such as facial recognition or e-commerce require models to learn new categories incrementally. Class-Incremental Learning (CIL) addresses this by updating models with new data without full retraining.

CIL faces two primary obstacles:

  1. Catastrophic Forgetting: As the model learns new classes, it tends to overwrite weights associated with old classes, leading to a sharp decline in performance on previously learned tasks.
  2. Poor Initialization: New classifiers are typically initialized randomly, ignoring the inherent semantic relationships between new and old categories (e.g., the visual similarity between a "cat" and a "tiger").

While existing methods often treat knowledge transfer as a one-way process (old to new), we argue that a bi-directional flow is necessary to maintain a balanced and accurate representation.

Methodology: The COIL Framework

The proposed COIL framework leverages Optimal Transport (OT) to create a semantic mapping between different learning stages. This mapping is used to facilitate two core components: Prospective Transport (PT) and Retrospective Transport (RT).

The COIL framework architecture showing Prospective and Retrospective Transport stages.

Prospective Transport (PT)

When new classes are introduced, COIL calculates the similarity between these new classes and stored exemplars of old classes within the feature space. Using the Sinkhorn algorithm, an optimal transport plan is derived to map the weights of established classifiers to the new ones. This provides a "warm start," allowing the model to leverage prior knowledge for faster and more accurate adaptation to new tasks.

Retrospective Transport (RT)

During the training phase for new classes, the same OT-based mapping is employed to transport knowledge from the newly learned classifiers back to the old ones. This serves as a sophisticated form of knowledge distillation. As the underlying feature space shifts to accommodate new data, RT ensures that the model’s understanding of old classes remains stable and aligned with the updated representations.

To ensure fairness between old and new classes, COIL utilizes a cosine classifier. By normalizing both features and weights, the model avoids biases toward recently seen data, which is a common pitfall in incremental learning.

Experimental Evaluation

COIL was evaluated on several standard benchmarks, including CIFAR-100, ImageNet-100/1000, and fine-grained datasets like CUB-200.

Key Results

COIL consistently achieved superior performance compared to state-of-the-art methods such as iCaRL, BiC, and WA.

Performance comparison on CIFAR-100 across different incremental task settings.
  • CIFAR-100: In a 20-step incremental setting, COIL reached a final accuracy of 62.98%, surpassing the next best method (BiC) by approximately 4%.
  • Fine-Grained Learning: On the CUB-200 dataset, particularly in "rare exemplar" scenarios (e.g., only 3 images per class), COIL outperformed BiC by 20%. This highlights its efficiency in memory-constrained environments.
  • Fast Adaptation: Decision boundary visualizations indicate that PT allows the model to classify new categories effectively even before explicit training, purely by leveraging semantic similarities.

Discussion and Insights

The success of COIL stems from its ability to translate visual similarity in the feature space into meaningful relationships between classifier weights. Unlike standard distillation, which primarily acts as a constraint, Retrospective Transport actively uses new information to reinforce old labels. This bi-directional flow creates a more robust and cohesive learning process.

Conclusion and Future Work

COIL represents a significant step forward in Class-Incremental Learning by effectively bridging the semantic gap through Optimal Transport. While highly effective, future research could focus on refining the estimation of marginal probabilities for class importance and exploring the application of semantic transport in completely memory-free (exemplar-free) settings. Additionally, optimizing the computational overhead of the Sinkhorn algorithm for extremely large-scale incremental steps remains an area for further development.

Summary updated

BibTeX
@inproceedings{zhou2021co,
  author = {Da-Wei Zhou and Han-Jia Ye and De-Chuan Zhan},
  title = {{Co-Transport for Class-Incremental Learning}},
  booktitle = {ACM Multimedia},
  year = {2021},
  doi = {10.1145/3474085.3475306},
}
Figure