Towards Realistic Incremental Scenario in Class Incremental Semantic Segmentation

Jihwan Kwak

Abstract

This paper addresses the unrealistic aspect of the commonly adopted Continuous Incremental Semantic Segmentation (CISS) scenario, termed overlapped. We point out that overlapped allows the same image to reappear in future tasks with different pixel labels, which is far from practical incremental learning scenarios. Moreover, we identified that this flawed scenario may lead to biased results for two commonly used techniques in CISS, pseudo-labeling and exemplar memory, resulting in unintended advantages or disadvantages for certain techniques. To mitigate this, a practical scenario called partitioned is proposed, in which the dataset is first divided into distinct subsets representing each class, and then the subsets are assigned to each corresponding task. This efficiently addresses the issue above while meeting the requirement of CISS scenario, such as capturing the background shifts. Furthermore, we identify and address the code implementation issues related to retrieving data from the exemplar memory, which was ignored in previous works. Lastly, we introduce a simple yet competitive memory-based baseline, MiB-AugM, that handles background shifts of current tasks in the exemplar memory. This baseline achieves state-of-the-art results across multiple tasks involving learning numerous new classes.

The Flaw in the Overlapped Scenario

In Class Incremental Semantic Segmentation (CISS), an AI model learns to identify new object categories over time without forgetting old ones. Currently, researchers use two main setups: disjoint and overlapped. The authors argue that the widely-used overlapped scenario is fundamentally unrealistic because it allows the same image to reappear across different tasks with conflicting labels.

Data reappearance and resulting issues in the standard overlapped CISS scenario.

As shown in the figure above, this creates two major issues:

  1. Unfair Advantage (Pseudo-labeling): Models using pseudo-labeling get an easy win because they have already seen the exact same image with the correct labels in a previous task. The authors' Pseudo-labeling Retrieval Rate (PRR) metric proved that models are significantly better at predicting labels for images they have seen before compared to truly "unseen" images.
  2. Label Conflict (Exemplar Memory): When using exemplar memory (saving old images to replay later), the model gets confused. It sees the same image twice: once where a pixel is labeled as a specific class (e.g., "cat") and once where it is labeled as "background."

Proposed Methodology

The Partitioned Scenario

To address these biases, the authors propose the partitioned scenario. Instead of letting images reappear, the dataset is first divided into distinct subsets based on the classes they contain. Each subset is assigned to only one task. This ensures no image is ever seen twice across different incremental steps, while still forcing the model to deal with "background shift"—where old or future classes are hidden in the background.

The proposed partitioned scenario ensures images are assigned to unique tasks, preventing reappearance.

Implementation Fixes and MiB-AugM

The authors discovered that several top-tier CISS codebases had a bug: when replaying images from memory, they were accidentally overwriting old ground-truth labels with "background" labels from the current task. After providing a corrected codebase, they introduced MiB-AugM, a simple but effective baseline. It combines the "Modeling the Background" (MiB) approach with a memory-based loss and uses a "prediction augmentation" technique to ensure that when the model sees a "background" pixel in a replayed image, it doesn't accidentally unlearn the new classes it just acquired.

Experimental Results

The researchers evaluated their methods using the Pascal VOC 2012 dataset across several incremental configurations (e.g., 15-1, 5-3, 10-5).

  • State-of-the-Art Performance: In complex tasks requiring high "plasticity" (like 5-3 and 10-5), MiB-AugM outperformed previous leaders like DKD-M by 4.94 and 2.21 mIoU points, respectively.
  • Resolving the Memory Paradox: In the "overlapped" scenario, adding memory sometimes hurt performance due to label conflicts. In the new "partitioned" scenario, adding memory consistently helps, proving that the conflict was an artifact of the experimental setup.
  • Simplicity: MiB-AugM only requires one hyperparameter ($\lambda$), making it much easier to tune than competing methods like PLOP or DKD.

Conclusion

By exposing the flaws in the "overlapped" scenario and introducing the "partitioned" alternative, this work provides a more rigorous framework for evaluating CISS. While currently tested on Pascal VOC, the authors plan to extend this logic to larger datasets like ADE20K. MiB-AugM stands as a robust new baseline that effectively balances the stability-plasticity dilemma in realistic incremental learning environments.

Summary updated

Abstract

This paper addresses the unrealistic aspect of the commonly adopted Continuous Incremental Semantic Segmentation (CISS) scenario, termed overlapped. We point out that overlapped allows the same image to reappear in future tasks with different pixel labels, which is far from practical incremental learning scenarios. Moreover, we identified that this flawed scenario may lead to biased results for two commonly used techniques in CISS, pseudo-labeling and exemplar memory, resulting in unintended advantages or disadvantages for certain techniques. To mitigate this, a practical scenario called partitioned is proposed, in which the dataset is first divided into distinct subsets representing each class, and then the subsets are assigned to each corresponding task. This efficiently addresses the issue above while meeting the requirement of CISS scenario, such as capturing the background shifts. Furthermore, we identify and address the code implementation issues related to retrieving data from the exemplar memory, which was ignored in previous works. Lastly, we introduce a simple yet competitive memory-based baseline, MiB-AugM, that handles background shifts of current tasks in the exemplar memory. This baseline achieves state-of-the-art results across multiple tasks involving learning numerous new classes.
BibTeX
@inproceedings{kwak2024towards,
  author = {Jihwan Kwak},
  title = {{Towards Realistic Incremental Scenario in Class Incremental Semantic Segmentation}},
  booktitle = {Third Conference on Lifelong Learning Agents},
  year = {2024},
  url = {https://openreview.net/forum?id=coPnMd4QaT},
}
Figure