Abstract
Despite the significant progress in continual image segmentation, existing arts still strive to balance between stability and plasticity. Additionally, they are specialist to specific tasks and models, which hinders the extension to more general situations. In this work, we present CUE, a novel Continual Universal sEgmentation pipeline that not only inherently tackles the stability-plasticity dilemma, but unifies any segmentation across tasks and models as well. Our key insight: any segmentation task can be reformulated as an understanding-then-refinement paradigm, which is inspired by humans' visual perception system to first perform high-level semantic understanding, then focus on low-level vision cues. We claim three desiderata for this design: Continuity by inherently avoiding the stability-plasticity dilemma via exploiting the natural differences between high-level and low-level knowledge; Generality by unifying and simplifying the landscape towards various segmentation tasks; and Efficiency as an interesting by-product by significantly reducing the research effort. Our resulting model shows significant improvements over previous state-of-the-arts across various segmentation tasks and datasets.
Introduction
Continual learning in computer vision aims to enable models to acquire new concepts incrementally without forgetting previously learned knowledge. However, image segmentation models frequently encounter the stability-plasticity dilemma: high plasticity leads to catastrophic forgetting of old classes, while high stability prevents effective learning of new data. Furthermore, most existing continual learning frameworks are "specialists," tailored for specific tasks like semantic segmentation, lacking the flexibility to generalize to instance or panoptic segmentation.
The CUE Pipeline: Decoupling "What" from "How"
To address these challenges, we propose CUE (Continual Universal sEgmentation). CUE is built on the principle of decoupling high-level semantic understanding from low-level visual refinement, a strategy inspired by human visual perception.
The pipeline consists of two primary stages:
- Semantic Understanding: A lightweight, DETR-like architecture generates class-aware "positional proposals." This module operates on low-resolution features, allowing for rapid adaptation to new classes.
- Visual Refinement: A frozen Segment Anything Model (SAM) serves as a refinement tool. By keeping the boundary-prediction module static, CUE ensures that the model's ability to produce precise masks remains constant, effectively "locking in" fragile low-level knowledge.
Knowledge Asymmetry and Stability
A key insight driving CUE is the observation of knowledge asymmetry. High-level semantic knowledge is relatively robust and easy for models to learn, whereas low-level boundary knowledge is fragile and converges slowly.
By delegating boundary refinement to a frozen expert (SAM), CUE avoids the catastrophic forgetting typically associated with fine-grained spatial details. This design also improves efficiency; CUE requires significantly fewer training epochs and achieves a much higher training throughput (63.5 FPS) compared to previous ViT-based methods.
Experimental Results
CUE was evaluated across multiple benchmarks, including PASCAL VOC 2012, ADE20K, COCO, and RefCOCOg, using the challenging "overlapped" continual learning protocol.
- ADE20K Panoptic Segmentation: CUE achieved 45.5 PQ, a substantial improvement over the previous SOTA of 33.9 PQ, reducing the performance gap to the joint-training baseline by over 34%.
- PASCAL VOC 2012: In an 11-step incremental setting, CUE maintained a 79.5 mIoU, significantly higher than the 64.1 mIoU of previous methods.
- Generality: The model successfully handled diverse tasks, including language-guided (referring) segmentation, demonstrating its universal applicability.
Conclusion
CUE represents a significant step toward universal continual segmentation. By reformulating segmentation as an understanding-then-refinement paradigm, it effectively balances stability and plasticity while simplifying the landscape of incremental learning across various vision tasks. Future work will explore scaling this pipeline to larger datasets and more complex multi-modal tasks.
Summary updated