Label Mapping Neural Networks with Response Consolidation for Class Incremental Learning

Xu Zhang, Yang Yao, Baile Xu, Lekun Mao, Furao Shen, Jian Zhao, Qingwei LinarXiv 2019 · OpenReview

TL;DR

This paper introduces LMRC, a novel approach to class incremental learning that tackles both catastrophic forgetting and a newly identified "softmax suppression" problem without needing to store old training data. LMRC uses a multi-head network with a unique "Label Mapping" strategy and "Response Consolidation" to effectively learn new classes while retaining knowledge of old ones.

What Problem Does This Paper Address?

Class incremental learning (CIL) is a crucial but challenging area in machine learning. It aims to enable models to learn new classes sequentially as data arrives, mimicking how humans learn. The primary hurdle in CIL is "catastrophic forgetting," where learning new information causes the model to lose previously acquired knowledge. Existing solutions often rely on storing or regenerating old data (rehearsal), which is inefficient in terms of space and computation, especially for streaming data where old data might be inaccessible.

This paper identifies a second major problem in CIL, particularly when rehearsal is not used: "softmax suppression." This occurs because when new classes are trained using the same output layer as old classes, the network is incentivized to assign higher probabilities to new classes, which in turn suppresses the probabilities of old classes, making them difficult to predict. This paper proposes a solution that addresses both catastrophic forgetting and softmax suppression simultaneously, without requiring access to old data.

Methods

The proposed model, Label Mapping with Response Consolidation (LMRC), introduces two key innovations:

  • Label Mapping (LM): To combat softmax suppression, LMRC replaces the traditional one-hot encoding for class labels with high-dimensional "label vectors." These vectors are generated using a novel algorithm that ensures sufficient distinctiveness between classes, even when they share the same output head. This approach allows for a multi-head network architecture where each head can specialize in a subset of classes, preventing direct competition in the softmax layer. The training objective for each head is to match its output to the corresponding label vector using cosine similarity.
  • Response Consolidation (RC): To address catastrophic forgetting without rehearsal, RC leverages the multi-head architecture. When learning new classes, the model first freezes the weights of the "old model" and propagates new data through its existing heads to capture their "response vectors." These response vectors then serve as targets for the corresponding heads in the "new model." This ensures that the behavior of the old heads is preserved, preventing the loss of old knowledge. The overall loss function combines the loss for new classes (using Label Mapping) and the Response Loss.

LMRC also offers an extension that can be combined with rehearsal methods if old data is available, further improving performance.

Datasets

The paper evaluates LMRC on two benchmark datasets:

  • CIFAR-100: A standard dataset for image classification.
  • ImageNet-200: A subset of ImageNet, also used for image classification.

These datasets are divided into sequential "class batches" to simulate the incremental learning scenario.

Key Insights and Ideas

  • Softmax Suppression is a distinct problem: The paper highlights that even if catastrophic forgetting is solved, softmax suppression remains a significant obstacle in CIL without rehearsal.
  • Multi-head networks are beneficial: The use of multiple output heads, each potentially responsible for a subset of classes, effectively decouples the learning of new and old classes, mitigating softmax suppression.
  • Label vectors offer better class discrimination: Replacing one-hot encoding with distinct label vectors in a high-dimensional space allows for more robust class representation and better separation between heads.
  • Response Consolidation effectively preserves old knowledge: By using the outputs of the frozen old model as targets for the new model's heads, LMRC successfully retains information about previously learned classes without needing the actual old data.
  • LMRC is highly efficient without rehearsal: The proposed method achieves state-of-the-art performance without relying on storing or regenerating old data, making it suitable for scenarios with limited memory or streaming data.

Key Results

  • LMRC significantly outperforms existing methods like Fine-tuning, EWC, and LwF.MC on both CIFAR-100 and ImageNet-200 datasets, achieving substantially higher average incremental accuracy. For instance, on ImageNet-200, LMRC achieved an average incremental accuracy of 66.4%, a ~40% improvement over other methods.
  • The effectiveness of both Label Mapping and Response Consolidation is validated through comparative experiments.
  • The Label Mapping algorithm demonstrates a high capacity for generating distinct label vectors, capable of supporting a large number of classes.
  • When extended with rehearsal, LMRC continues to achieve the best performance and shows a greater advantage with fewer samples of old classes, indicating its efficiency.

Limitations and Future Work

  • Confusion Problem: While Label Mapping mitigates the confusion problem in multi-head networks (where it's unclear which head to use for prediction), the paper acknowledges that it can still become more significant as the number of heads and classes per head increases. The Winner-Take-All strategy is used to address this during testing.
  • Theoretical Analysis of Label Mapping: The paper provides a theoretical analysis of the capacity of Label Mapping, showing that it can generate a large number of distinct label vectors, which is crucial for scalability. However, the practical implications of very large numbers of classes and the trade-offs between dimensionality, threshold, and capacity could be further explored.
  • Future Work: The paper suggests that LMRC can be combined with rehearsal methods, implying that further research could explore optimal ways to integrate these approaches. The scalability to even larger and more complex datasets could also be a direction for future work.

Summary updated

Abstract

Class incremental learning refers to a special multi-class classification task, in which the number of classes is not fixed but is increasing with the continual arrival of new data. Existing researches mainly focused on solving catastrophic forgetting problem in class incremental learning. To this end, however, these models still require the old classes cached in the auxiliary data structure or models, which is inefficient in space or time. In this paper, it is the first time to discuss the difficulty without support of old classes in class incremental learning, which is called as softmax suppression problem. To address these challenges, we develop a new model named Label Mapping with Response Consolidation (LMRC), which need not access the old classes anymore. We propose the Label Mapping algorithm combined with the multi-head neural network for mitigating the softmax suppression problem, and propose the Response Consolidation method to overcome the catastrophic forgetting problem. Experimental results on the benchmark datasets show that our proposed method achieves much better performance compared to the related methods in different scenarios.
BibTeX
@misc{zhang2019label,
  author = {Xu Zhang and Yang Yao and Baile Xu and Lekun Mao and Furao Shen and Jian Zhao and Qingwei Lin},
  title = {{Label Mapping Neural Networks with Response Consolidation for Class Incremental Learning}},
  year = {2019},
  eprint = {1905.07835},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1905.07835},
}
Figure