TL;DR
This paper introduces "Sparsely Supervised Instance Segmentation" (SpSIS), a new annotation paradigm where only bounding boxes and a few instance masks are provided. They propose MaskBooster, an end-to-end self-training framework that dynamically generates and refines pseudo masks, significantly boosting performance on SpSIS tasks and outperforming existing weakly and semi-supervised methods.
What Problem Does This Paper Address?
Instance segmentation, a task that involves identifying and outlining individual objects in an image, typically requires a large amount of meticulously annotated data, specifically pixel-level masks for each object. This annotation process is time-consuming and expensive, limiting the practical application of instance segmentation. To address this, researchers have explored various forms of "incomplete supervision," such as using only bounding boxes (Weakly Supervised Instance Segmentation - WSIS) or a mix of fully annotated and unannotated images (Semi-Supervised Instance Segmentation - SSIS).
This paper introduces a new scenario called "Sparsely Supervised Instance Segmentation" (SpSIS). In SpSIS, datasets provide bounding boxes for all objects, but only a small fraction of these objects also have ground-truth masks. This is a more realistic scenario than WSIS, as annotating a few masks is more feasible than annotating all of them, but still presents a challenge for standard instance segmentation models. The core problem is how to effectively train instance segmentation models with this limited mask supervision, leveraging the abundant bounding box information and the few available masks.
Methods
The paper proposes MaskBooster, an end-to-end self-training framework designed for SpSIS. It operates on a teacher-student architecture:
- Teacher-Student Architecture with EMA: A "teacher" model, updated via Exponential Moving Average (EMA) of the student's weights, generates pseudo masks. The "student" model is trained using these pseudo masks and ground truth data. This mutual learning process allows the teacher to improve over time, leading to better pseudo masks and thus a better student.
- Dynamic Pseudo Masks: Unlike previous methods that used static pseudo masks, MaskBooster generates and updates pseudo masks dynamically during training. This avoids tedious multi-stage training and allows the model to adapt.
- Pseudo Mask Loss Reweighting: To prevent model collapse when ground-truth masks are extremely scarce, MaskBooster employs a re-weighting strategy for the pseudo mask loss. This factor
r(t)gradually increases the influence of pseudo mask loss as training progresses. - T-S Matching: To speed up training and improve efficiency, a "T-S Matching" strategy is used. This ensures that pseudo masks generated by the teacher directly correspond to the student's predictions, eliminating the need for complex GT assignment.
- Box Prior for Binary Mask Refining: To improve the quality of binary pseudo masks, a "box prior" is used. Pixels predicted as foreground by the teacher but falling outside the ground-truth bounding box are forced to be background. This leverages the accurate localization information from bounding boxes.
- Inter-Class Relationship for Soft Pseudo Masks: The paper observes an inherent inter-class relationship in mask logits (the raw output before the final prediction). To exploit this, they introduce "soft pseudo masks" derived from these logits. Knowledge distillation is then applied to learn this inter-class relationship implicitly, using Kullback-Leibler divergence between the teacher's and student's probability distributions of mask logits.
- Wrapper Functionality: MaskBooster is designed as a universal wrapper that can be applied to most existing instance segmentation algorithms, converting them into SpSIS-capable models.
Datasets
The paper uses the following datasets for training and evaluation:
- COCO: Standard COCO dataset is used to create SpSIS protocols with varying ratios of ground-truth masks:
- COCO 0.1% (approx. 880 GT masks)
- COCO 1% (approx. 8656 GT masks)
- COCO 10% (approx. 86k GT masks)
- BDD100K: A dataset commonly used for autonomous driving scenarios, also used to evaluate MaskBooster's performance on different domains.
Key Insights and Ideas
- SpSIS is a practical and challenging scenario: The paper effectively defines and motivates SpSIS as a more realistic and cost-effective annotation paradigm for instance segmentation.
- Dynamic pseudo masks are crucial for self-training: Moving from static to dynamic pseudo masks significantly improves the self-training process, allowing for continuous adaptation and better convergence.
- Leveraging bounding boxes beyond localization: The "box prior" method demonstrates that bounding boxes can be used not just for localization but also to refine the quality of generated masks, preventing erroneous foreground predictions outside the box.
- Inter-class relationships hold valuable information: The observation that mask logits from similar categories share information is a key insight. Exploiting this through knowledge distillation for "soft pseudo masks" allows the model to learn richer representations than binary masks alone.
- MaskBooster is a versatile tool: Its design as a wrapper makes it applicable to various existing instance segmentation architectures, offering a straightforward way to improve their performance in SpSIS settings.
Key Results
- Significant performance gains on SpSIS: MaskBooster consistently outperforms sparsely supervised baselines across different SpSIS protocols (COCO 0.1%, 1%, 10%) and base models (Mask RCNN, ShapeProp). For example, on COCO 0.1%, it achieves +5.7 AP higher than the baseline Mask RCNN.
- Closing the gap with fully supervised methods: On COCO 10%, MaskBooster narrows the performance gap between sparsely supervised methods and fully supervised ones to as little as 0.9-1.7 AP.
- Outperforms WSIS and SSIS: MaskBooster, when applied to SpSIS datasets, demonstrates superior performance compared to state-of-the-art WSIS and SSIS algorithms under similar annotation budgets. For instance, it achieves +1.8 AP higher than BoxInst (WSIS) with only a small increase in annotation budget.
- Effective on diverse datasets: The method shows strong performance on the BDD100K dataset, indicating its robustness across different domains.
- Ablation studies confirm component effectiveness: Experiments show that the EMA+Aug strategy provides some improvement, but the optimized binary pseudo masks (with box prior) and soft pseudo masks (with knowledge distillation) contribute significantly to the overall performance boost.
Limitations and Future Work
- Limitations:
- The paper acknowledges that when GT masks are very limited (e.g., 0.1%), the model's performance is still significantly lower than fully supervised methods, highlighting the inherent challenge of extreme sparsity.
- While MaskBooster improves performance, it doesn't fully bridge the gap to fully supervised methods in the most extreme SpSIS scenarios.
- Future Work:
- The authors suggest investigating more universal supervision paradigms that involve incomplete annotations for both bounding boxes and masks.
- They believe their method has the potential to address such more complex scenarios.
Summary updated