Abstract
Few-shot semantic segmentation typically relies on expensive pixel-level masks for meta-training. This paper proposes a framework to train these models using only bounding box annotations, which are significantly cheaper to obtain. The core of the approach is a novel Trimap Attention Module (TAM) and a Pseudo Trimap Estimator that together "purify" the noisy signals within bounding boxes. Experimental results on Pascal-5i, FSS-1000, and VOC2COCO demonstrate that this method achieves performance comparable to mask-supervised models while being much more label-efficient.
Introduction
Semantic segmentation—the task of labeling every pixel in an image—is notoriously data-hungry. While Few-Shot Learning (FSL) aims to segment new objects using only a few examples (the "support set"), there is a hidden cost: to reach that level of flexibility, models must first undergo "meta-training" on a massive dataset of other objects. Traditionally, this meta-training requires full pixel-level masks.
The authors point out a paradox: we use few-shot learning to be "label-efficient," yet we still require thousands of manual pixel-level masks to train the learner in the first place. Bounding boxes are a much cheaper alternative, taking seconds to produce compared to over an hour for complex masks. However, boxes are "impure" as they contain both the object and distracting background. Simply feeding boxes into existing few-shot algorithms causes performance to degrade because the model becomes confused by background pixels inside the box.
Methodology
The proposed meta-learning framework is designed to "purify" the noisy signals coming from bounding boxes. The architecture consists of two main components integrated into a standard few-shot segmentation pipeline.
Pseudo Trimap Estimator
Instead of treating the entire bounding box as the object, the model estimates a "trimap." This divides the image into three zones: confident foreground, confident background, and an "uncertain" (gray) region. By logically subtracting the predicted mask from the box mask, the model identifies which pixels are likely "false negatives" (background pixels inside the box) and ignores them during training.
Trimap Attention Module (TAM)
The TAM is an iterative refinement system that uses "T-Attention" blocks to alternate roles between the support image (the example) and the query image (the target).
As shown in the figure above, the module performs the following:
- It uses the query's current prediction to refine the support's "prototype" (the mathematical representation of the object).
- It then uses that cleaner prototype to improve the query's segmentation.
- This iterative "conversation" between images helps the model converge on a much more accurate representation of the object, effectively denoising the bounding box input.
Experimental Setup
The authors evaluated their method on three major benchmarks:
- Pascal-5ⁱ: A standard few-shot segmentation dataset derived from Pascal VOC 2012.
- FSS-1000: A large-scale dataset containing 1,000 different object classes, providing a rigorous test of generalization.
- VOC2COCO: A cross-dataset setup where the model is trained on Pascal VOC and tested on MS-COCO to evaluate robustness to domain shifts.
Key Results and Insights
The proposed method consistently outperformed baselines trained on bounding boxes. On Pascal-5ⁱ, in a 1-way 5-shot setting, the model gained 1.4% mean-IoU when masks were available at test time, and a significant 3.6% mean-IoU when only boxes were available at test time. On FSS-1000, the method recovered more than 3% P-IoU on unseen classes compared to the baseline, proving its scalability.
The primary insight is that the failure of box-supervised models is largely due to "contamination" of the object prototype by background pixels. Iterative refinement is key; by updating the prototype and the prediction in cycles, the model gradually "shrinks" its focus onto the actual object of interest.
Conclusion
This work demonstrates that bounding boxes are a "Goldilocks" compromise for labels—providing enough spatial information to guide segmentation while remaining highly efficient to produce. While a small performance gap remains between box-supervision and mask-supervision, the TAM module significantly closes it. Future work may explore even weaker forms of supervision, such as points or scribbles, during the meta-training phase.
Summary updated