TL;DR
This paper tackles the challenge of continual semantic segmentation (CSS), where models must learn new object classes over time without forgetting previously learned ones. The authors propose a novel approach that leverages cheaper image-level labels for past classes and introduces innovative rehearsal strategies that blend data augmentation and knowledge distillation, significantly improving performance and reducing catastrophic forgetting.
What Problem Does This Paper Address?
Semantic segmentation, a crucial task for understanding images, typically requires large datasets with pixel-level annotations for every object. However, in real-world applications, new object classes are frequently introduced, necessitating models that can adapt incrementally. This leads to the problem of "catastrophic forgetting," where learning new classes erases knowledge of old ones. A specific challenge in CSS is "background shift," where previously learned object classes become part of the background for new classes, amplifying forgetting. Existing solutions often rely on expensive pixel-level annotations for all data or require large amounts of auxiliary data. This paper addresses the need for a more data-efficient CSS approach that can effectively utilize cheaper image-level labels for past classes and introduce novel rehearsal techniques.
Methods
The core of the proposed method lies in a weakly-supervised mechanism for CSS. This involves:
- Grad-CAM and Self-Attention for Pseudo-Label Generation: The model uses Grad-CAM to generate coarse localization masks from image-level labels of past classes. These masks are then refined by a self-attention module, which learns to propagate information and improve localization. This refined output is used to generate pseudo-labels for training.
- Consistency Loss: A consistency loss is employed to ensure that the model's predictions on strongly augmented images are consistent with the generated pseudo-labels, enabling self-training.
- Novel Rehearsal Strategies: Two variants of a rehearsal strategy are introduced, blending data augmentation and knowledge distillation:
- Image-based Rehearsal (Copy-Paste): Pixels from past objects stored in a memory bank are randomly pasted onto new training images.
- Feature-based Rehearsal: Intermediate feature representations of past classes are stored and injected into the current network's feature maps. This approach offers privacy benefits as it doesn't store raw pixels.
- Knowledge Distillation: Standard knowledge distillation losses are used to further prevent forgetting by regularizing the outputs of the current model with those of the previous one.
The overall loss function combines supervised loss, weakly-supervised loss, classification loss (for CAM generation), and knowledge distillation loss.
Datasets
The experiments were conducted on the PASCAL-VOC 2012 dataset. The paper uses standard splits and simulates continual learning scenarios by dividing the 20 semantic classes into different incremental steps (e.g., 19-1, 15-5, 15-1).
Key Insights and Ideas
- Leveraging Weak Labels is Powerful: The paper demonstrates that even cheap image-level labels for past classes can significantly improve performance in CSS, reducing forgetting and boosting performance on both old and new classes. This is a crucial finding for practical applications where pixel-level annotations are scarce.
- Less Data Can Be Better (Sometimes): Surprisingly, the authors found that in some scenarios, models trained with a reduced proportion of fully-supervised data (and more weakly-supervised data) achieved better results than those trained with 100% fully-supervised data. This suggests that reducing the amount of "noisy" or conflicting data might help find better minima that are closer to previous solutions, mitigating forgetting.
- Feature-based Rehearsal Offers Privacy and Performance: The feature-based rehearsal strategy is highlighted for its privacy advantages (no raw pixels stored) and its competitive, sometimes superior, performance compared to image-based rehearsal. This is particularly relevant in applications with strict data privacy requirements.
- Rehearsal as Data Augmentation: The novel framing of rehearsal as a data augmentation technique, particularly the feature-based approach, is a key conceptual contribution. It shows how to integrate past knowledge in a more sophisticated way than simple replay.
Key Results
- The proposed approach consistently improves the mean Intersection-over-Union (mIoU) on both old and new classes across various continual learning scenarios (19-1, 15-5, 15-1).
- In the 19-1 overlapped setting, the proposed method achieved 72.3% on old classes and 31.0% on novel classes, outperforming the MiB baseline (70.2% and 22.1%, respectively) when using weak supervision for old classes.
- With a small budget of just 20 examples per old class, the rehearsal techniques significantly improved mIoU.
- In scenarios with reduced fully-supervised data, the proposed method and the MiB baseline showed improved performance on novel classes compared to using full supervision, with the proposed method often achieving better results. For instance, in the 15-5 disjoint setup with half the data, MiB achieved 71.8%/44.9% (old/new) compared to 71.8%/43.3% with full supervision.
Limitations and Future Work
- 15-1 Scenario Challenge: The paper notes that in scenarios where classes are learned one by one (15-1), stronger knowledge distillation methods (like PLOP) and metric learning approaches (like SDR) sometimes outperform their method. The authors suggest that future work should investigate combining their weakly-supervised mechanisms and rehearsal with these other strategies.
- Hypotheses for "Less Data is Better": The paper offers hypotheses for why less data might be beneficial but acknowledges this requires further investigation.
- Few-Shot Learning and Meta-Learning: The authors suggest that exploring few-shot learning and meta-learning strategies could be a promising direction for improving CSS, especially in light of the observed benefits of reduced data.
- Investigating Other Rehearsal Modules: While feature-based rehearsal from the 3rd ResNet module showed good results, exploring features from other layers or different rehearsal mechanisms could be beneficial.
Summary updated