Inherit With Distillation and Evolve With Contrast: Exploring Class Incremental Semantic Segmentation Without Exemplar Memory

Danpei Zhao, Bo Yuan, Zhenwei Shi

TL;DR

This paper introduces IDEC, a novel method for Class Incremental Semantic Segmentation (CISS) that tackles catastrophic forgetting and semantic drift without relying on storing past data (exemplars). IDEC combines a dense knowledge distillation approach with asymmetric contrastive learning to effectively learn new classes while retaining knowledge of old ones, achieving state-of-the-art results on challenging benchmarks.

What Problem Does This Paper Address?

Semantic segmentation, a task that assigns a label to every pixel in an image, typically requires large, fully labeled datasets. However, in real-world scenarios, data often arrives incrementally. Class Incremental Semantic Segmentation (CISS) aims to train models on a sequence of tasks, where each task introduces new classes. The primary challenges in CISS are:

  • Catastrophic Forgetting: When a model learns new classes, it tends to forget previously learned ones.
  • Semantic Drift: The meaning of background pixels can change as new classes are introduced, leading to confusion and misclassification.
  • Data Scarcity: Traditional CISS methods often rely on storing "exemplars" (a subset of past data) to mitigate forgetting. However, this is not always feasible due to storage limitations or privacy concerns.

Existing methods often struggle to balance learning new information with preserving old knowledge, leading to performance degradation. This paper specifically addresses CISS in an "exemplar-free" setting, meaning it doesn't store past data.

Methods

The proposed IDEC (Inherit with Distillation and Evolve with Contrast) method comprises two main components:

  1. Dense Knowledge Distillation on all Aspects (DADA):
    • This module aims to transfer knowledge from the previous model ($M_{t-1}$) to the current model ($M_t$).
  2. Unlike previous methods that might focus only on output logits, DADA distills knowledge from intermediate layers as well as the final output logits.
  3. It emphasizes "semantic-invariant" knowledge, which is believed to be more robust and crucial for retaining old class information.
  4. An Attenuated Layer-aware Weight (ALW) is introduced to dynamically adjust the importance of distillation across different layers, giving more weight to deeper layers that capture more semantic information.
  5. This distillation is performed synchronously with the training of the current model.
  6. Asymmetric Region-wise Contrastive Learning (ARCL):
    • This module tackles semantic drift and classifier bias.
  7. It operates in the "latent space" (feature representations) of the models.
  8. The key idea is "asymmetric" contrastive learning: it uses embeddings from the old model ($M_{t-1}$) as anchors and embeddings from the current model ($M_t$) as positive and negative examples. This is crucial because the old model's representations are more stable for past classes.
  9. It performs contrastive learning on a region-wise basis, comparing groups of pixels belonging to the same class, rather than just individual pixels. This helps capture richer context.
  10. A Dynamic Class-specific Pseudo-Labelling (DCPL) strategy is also employed to generate pseudo-labels for old classes using the previous model, further aiding in supervision.

These two modules work together in an end-to-end fashion.

Datasets

The proposed method was evaluated on the following standard datasets for semantic segmentation:

  • Pascal VOC 2012: A widely used benchmark with 20 semantic classes.
  • ADE20K: A larger dataset with 150 classes, covering a wider variety of indoor and outdoor scenes.
  • ISPRS: Airborne image datasets (Postdam and Vaihingen) for remote-sensing applications.

The experiments were conducted across various incremental settings (e.g., 15-5, 15-1, 100-10, 2-1) to simulate different learning scenarios.

Key Insights and Ideas

  • Synergistic Forgetting and Drift Mitigation: The paper argues that catastrophic forgetting and semantic drift are intertwined problems in CISS and proposes a unified solution.
  • Importance of Intermediate Layer Distillation: Distilling knowledge from intermediate layers, not just the final output, is crucial for effective knowledge transfer in dense prediction tasks like segmentation.
  • Asymmetric Contrastive Learning for Stability: Using the older, more stable model's representations as anchors in contrastive learning provides a more reliable reference point for learning new classes and preventing drift.
  • Region-wise Contrast for Context: Focusing contrastive learning on regions of pixels, rather than individual pixels, better captures the contextual information vital for semantic segmentation.
  • Exemplar-Free is Achievable and Effective: The work demonstrates that state-of-the-art performance can be achieved in CISS without the need to store past data, addressing practical limitations.

Key Results

  • State-of-the-Art Performance: IDEC achieved state-of-the-art results on Pascal VOC 2012, ADE20K, and ISPRS datasets across various incremental learning settings.
  • Superior Anti-Forgettting: In multi-step CISS tasks, IDEC showed significantly better performance in retaining knowledge of old classes compared to existing methods like PLOP and RCIL. For instance, on VOC 15-1, IDEC achieved 67.32% mIoU for all classes, outperforming the closest contender.
  • Robustness to Incremental Order: The method demonstrated stable performance even with different orders of class introduction, indicating robustness.
  • Effective Module Contributions: Ablation studies confirmed the significant contributions of both the DADA distillation and ARCL contrastive learning modules, as well as the DCPL strategy.

Limitations and Future Work

  • Computational Cost: The paper acknowledges that the contrastive learning module (ARCL) can be computationally expensive, particularly with a large number of anchor classes. Future work might explore strategies to alleviate this, such as limiting the number of anchor classes.
  • Segmentation Model Impact: While the paper investigates the impact of different segmentation backbones (e.g., DeepLabv3, Swin-Transformer), it suggests that employing a more robust segmentation model can further boost CISS performance.
  • Future Directions: The authors suggest exploring human-like anti-forgetting mechanisms, adapting the method to more complex scenarios like open-set recognition, and investigating few-shot/zero-shot CISS.

Summary updated

Abstract

As a front-burner problem in incremental learning, class incremental semantic segmentation (CISS) is plagued by catastrophic forgetting and semantic drift. Although recent methods have utilized knowledge distillation to transfer knowledge from the old model, they are still unable to avoid pixel confusion, which results in severe misclassification after incremental steps due to the lack of annotations for past and future classes. Meanwhile data-replay-based approaches suffer from storage burdens and privacy concerns. In this paper, we propose to address CISS without exemplar memory and resolve catastrophic forgetting as well as semantic drift synchronously. We present Inherit with Distillation and Evolve with Contrast (IDEC), which consists of a Dense Knowledge Distillation on all Aspects (DADA) manner and an Asymmetric Region-wise Contrastive Learning (ARCL) module. Driven by the devised dynamic class-specific pseudo-labelling strategy, DADA distils intermediate-layer features and output-logits collaboratively with more emphasis on semantic-invariant knowledge inheritance. ARCL implements region-wise contrastive learning in the latent space to resolve semantic drift among known classes, current classes, and unknown classes. We demonstrate the effectiveness of our method on multiple CISS tasks by state-of-the-art performance, including Pascal VOC 2012, ADE20K and ISPRS datasets. Our method also shows superior anti-forgetting ability, particularly in multi-step CISS tasks.
BibTeX
@article{zhao2023inherit,
  author = {Danpei Zhao and Bo Yuan and Zhenwei Shi},
  title = {{Inherit With Distillation and Evolve With Contrast: Exploring Class Incremental Semantic Segmentation Without Exemplar Memory}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
  url = {https://openreview.net/forum?id=GjAyqkgT3DW},
}
Figure