Abstract
Few-shot semantic segmentation (FSS) aims to segment novel object categories using only a handful of annotated examples. A primary challenge in FSS is the creation of representative class prototypes from limited data, which often leads to biased representations. This paper introduces PAIA (Prototype Augmentation with Image-level Annotations), a novel framework that enhances class prototypes by incorporating easily obtainable image-level weak annotations. By utilizing a Distilled Soft-Masked Average Pooling (DSMP) technique and an Iterative Fusion Module (IFM), PAIA effectively filters background noise from auxiliary images and refines class features. Experimental results on PASCAL-$5^i$ and COCO-$20^i$ demonstrate that PAIA achieves state-of-the-art performance, significantly outperforming existing baselines with minimal additional annotation effort.
Introduction
Semantic segmentation typically requires dense, pixel-level annotations, which are costly and time-consuming to produce. Few-shot semantic segmentation (FSS) mitigates this by learning to generalize from a small "support set" of labeled images. However, the resulting "prototypes"—latent representations of a class—are often inaccurate due to the scarcity of data.
While auxiliary image-level annotations (tags indicating presence without location) are inexpensive, they are difficult to integrate because they lack spatial precision and often contain distracting background objects. PAIA addresses this by distilling useful class information from these weak labels while suppressing noise, thereby enriching the prototype's discriminative power.
Methodology
The PAIA framework builds robust class prototypes in a metric space by fusing strong pixel-level data from the support set with weak image-level data from an auxiliary set.
Distilled Soft-Masked Average Pooling (DSMP)
Standard methods often use hard binary masks for pseudo-labeling, which can propagate errors. PAIA employs Soft-Masked Average Pooling (SMP), assigning continuous confidence weights to pixels. To further refine this, the Distilled SMP (DSMP) module applies a task-specific threshold based on the statistical distribution of feature distances (variance, skewness, etc.) between pixels and the initial prototype. This allows the model to dynamically filter out uncertain background regions in auxiliary images.
Iterative Fusion Module (IFM)
The Iterative Fusion Module (IFM) serves as the core refinement engine. It iteratively updates the fused prototype by re-evaluating the probability maps of auxiliary images. This process gradually pulls in more accurate class-related features while reducing the initial bias inherent in the small support set.
Experimental Setup
The proposed model was evaluated on two standard benchmarks:
- PASCAL-$5^i$: Derived from PASCAL VOC 2012, with 20 categories divided into four folds for cross-validation.
- COCO-$20^i$: A more challenging dataset built from MS COCO, featuring 80 categories across four folds. In both cases, unsegmented images with category tags were utilized as the auxiliary set to simulate low-cost data augmentation.
Results and Discussion
PAIA achieves state-of-the-art results across both benchmarks. On PASCAL-$5^i$, using a ResNet-50 backbone, the model reached a mean Intersection over Union (mIoU) of 54.03% in the 1-shot setting and 64.04% in the 5-shot setting.
Key findings include:
- Efficiency: PAIA outperformed the PPNet baseline by over 1% mIoU while requiring fewer auxiliary images (2 vs. 3).
- Robustness: The framework remains effective even when support masks are replaced with extremely weak annotations like scribbles or bounding boxes, showing improvements of up to 8.2% over baselines.
- Noise Suppression: The DSMP module proved critical in preventing "background noise" (e.g., a car in the background of a "dog" auxiliary image) from corrupting the class prototype.
Conclusion
PAIA demonstrates that image-level weak annotations are a powerful resource for few-shot semantic segmentation when paired with effective distillation and fusion mechanisms. By dynamically filtering noise and iteratively refining prototypes, PAIA sets a new standard for performance and annotation efficiency in the field. Future research could explore the application of these statistical filtering techniques to fully unsupervised or panoptic segmentation tasks.
Summary updated