TL;DR
This paper introduces IDEC, a novel method for Class Incremental Semantic Segmentation (CISS) that tackles catastrophic forgetting and semantic drift without relying on storing past data (exemplars). IDEC combines a dense knowledge distillation approach with asymmetric contrastive learning to effectively learn new classes while retaining knowledge of old ones, achieving state-of-the-art results on challenging benchmarks.
What Problem Does This Paper Address?
Semantic segmentation, a task that assigns a label to every pixel in an image, typically requires large, fully labeled datasets. However, in real-world scenarios, data often arrives incrementally. Class Incremental Semantic Segmentation (CISS) aims to train models on a sequence of tasks, where each task introduces new classes. The primary challenges in CISS are:
- Catastrophic Forgetting: When a model learns new classes, it tends to forget previously learned ones.
- Semantic Drift: The meaning of background pixels can change as new classes are introduced, leading to confusion and misclassification.
- Data Scarcity: Traditional CISS methods often rely on storing "exemplars" (a subset of past data) to mitigate forgetting. However, this is not always feasible due to storage limitations or privacy concerns.
Existing methods often struggle to balance learning new information with preserving old knowledge, leading to performance degradation. This paper specifically addresses CISS in an "exemplar-free" setting, meaning it doesn't store past data.
Methods
The proposed IDEC (Inherit with Distillation and Evolve with Contrast) method comprises two main components:
- Dense Knowledge Distillation on all Aspects (DADA):
- This module aims to transfer knowledge from the previous model ($M_{t-1}$) to the current model ($M_t$).
- Unlike previous methods that might focus only on output logits, DADA distills knowledge from intermediate layers as well as the final output logits.
- It emphasizes "semantic-invariant" knowledge, which is believed to be more robust and crucial for retaining old class information.
- An Attenuated Layer-aware Weight (ALW) is introduced to dynamically adjust the importance of distillation across different layers, giving more weight to deeper layers that capture more semantic information.
- This distillation is performed synchronously with the training of the current model.
- Asymmetric Region-wise Contrastive Learning (ARCL):
- This module tackles semantic drift and classifier bias.
- It operates in the "latent space" (feature representations) of the models.
- The key idea is "asymmetric" contrastive learning: it uses embeddings from the old model ($M_{t-1}$) as anchors and embeddings from the current model ($M_t$) as positive and negative examples. This is crucial because the old model's representations are more stable for past classes.
- It performs contrastive learning on a region-wise basis, comparing groups of pixels belonging to the same class, rather than just individual pixels. This helps capture richer context.
- A Dynamic Class-specific Pseudo-Labelling (DCPL) strategy is also employed to generate pseudo-labels for old classes using the previous model, further aiding in supervision.
These two modules work together in an end-to-end fashion.
Datasets
The proposed method was evaluated on the following standard datasets for semantic segmentation:
- Pascal VOC 2012: A widely used benchmark with 20 semantic classes.
- ADE20K: A larger dataset with 150 classes, covering a wider variety of indoor and outdoor scenes.
- ISPRS: Airborne image datasets (Postdam and Vaihingen) for remote-sensing applications.
The experiments were conducted across various incremental settings (e.g., 15-5, 15-1, 100-10, 2-1) to simulate different learning scenarios.
Key Insights and Ideas
- Synergistic Forgetting and Drift Mitigation: The paper argues that catastrophic forgetting and semantic drift are intertwined problems in CISS and proposes a unified solution.
- Importance of Intermediate Layer Distillation: Distilling knowledge from intermediate layers, not just the final output, is crucial for effective knowledge transfer in dense prediction tasks like segmentation.
- Asymmetric Contrastive Learning for Stability: Using the older, more stable model's representations as anchors in contrastive learning provides a more reliable reference point for learning new classes and preventing drift.
- Region-wise Contrast for Context: Focusing contrastive learning on regions of pixels, rather than individual pixels, better captures the contextual information vital for semantic segmentation.
- Exemplar-Free is Achievable and Effective: The work demonstrates that state-of-the-art performance can be achieved in CISS without the need to store past data, addressing practical limitations.
Key Results
- State-of-the-Art Performance: IDEC achieved state-of-the-art results on Pascal VOC 2012, ADE20K, and ISPRS datasets across various incremental learning settings.
- Superior Anti-Forgettting: In multi-step CISS tasks, IDEC showed significantly better performance in retaining knowledge of old classes compared to existing methods like PLOP and RCIL. For instance, on VOC 15-1, IDEC achieved 67.32% mIoU for all classes, outperforming the closest contender.
- Robustness to Incremental Order: The method demonstrated stable performance even with different orders of class introduction, indicating robustness.
- Effective Module Contributions: Ablation studies confirmed the significant contributions of both the DADA distillation and ARCL contrastive learning modules, as well as the DCPL strategy.
Limitations and Future Work
- Computational Cost: The paper acknowledges that the contrastive learning module (ARCL) can be computationally expensive, particularly with a large number of anchor classes. Future work might explore strategies to alleviate this, such as limiting the number of anchor classes.
- Segmentation Model Impact: While the paper investigates the impact of different segmentation backbones (e.g., DeepLabv3, Swin-Transformer), it suggests that employing a more robust segmentation model can further boost CISS performance.
- Future Directions: The authors suggest exploring human-like anti-forgetting mechanisms, adapting the method to more complex scenarios like open-set recognition, and investigating few-shot/zero-shot CISS.
Summary updated