Few-Shot Semantic Segmentation Augmented with Image-Level Weak Annotations

Shuo Lei, Xuchao Zhang, Jianfeng He, Fanglan Chen, Chang-Tien Lu

Abstract

Few-shot semantic segmentation (FSS) aims to segment novel object categories using only a handful of annotated examples. A primary challenge in FSS is the creation of representative class prototypes from limited data, which often leads to biased representations. This paper introduces PAIA (Prototype Augmentation with Image-level Annotations), a novel framework that enhances class prototypes by incorporating easily obtainable image-level weak annotations. By utilizing a Distilled Soft-Masked Average Pooling (DSMP) technique and an Iterative Fusion Module (IFM), PAIA effectively filters background noise from auxiliary images and refines class features. Experimental results on PASCAL-$5^i$ and COCO-$20^i$ demonstrate that PAIA achieves state-of-the-art performance, significantly outperforming existing baselines with minimal additional annotation effort.

Introduction

Semantic segmentation typically requires dense, pixel-level annotations, which are costly and time-consuming to produce. Few-shot semantic segmentation (FSS) mitigates this by learning to generalize from a small "support set" of labeled images. However, the resulting "prototypes"—latent representations of a class—are often inaccurate due to the scarcity of data.

While auxiliary image-level annotations (tags indicating presence without location) are inexpensive, they are difficult to integrate because they lack spatial precision and often contain distracting background objects. PAIA addresses this by distilling useful class information from these weak labels while suppressing noise, thereby enriching the prototype's discriminative power.

Methodology

The PAIA framework builds robust class prototypes in a metric space by fusing strong pixel-level data from the support set with weak image-level data from an auxiliary set.

Distilled Soft-Masked Average Pooling (DSMP)

Standard methods often use hard binary masks for pseudo-labeling, which can propagate errors. PAIA employs Soft-Masked Average Pooling (SMP), assigning continuous confidence weights to pixels. To further refine this, the Distilled SMP (DSMP) module applies a task-specific threshold based on the statistical distribution of feature distances (variance, skewness, etc.) between pixels and the initial prototype. This allows the model to dynamically filter out uncertain background regions in auxiliary images.

Iterative Fusion Module (IFM)

The Iterative Fusion Module (IFM) serves as the core refinement engine. It iteratively updates the fused prototype by re-evaluating the probability maps of auxiliary images. This process gradually pulls in more accurate class-related features while reducing the initial bias inherent in the small support set.

The PAIA architecture illustrating the feature extraction process and the Iterative Fusion Model (IFM) for prototype refinement.

Experimental Setup

The proposed model was evaluated on two standard benchmarks:

  • PASCAL-$5^i$: Derived from PASCAL VOC 2012, with 20 categories divided into four folds for cross-validation.
  • COCO-$20^i$: A more challenging dataset built from MS COCO, featuring 80 categories across four folds. In both cases, unsegmented images with category tags were utilized as the auxiliary set to simulate low-cost data augmentation.

Results and Discussion

PAIA achieves state-of-the-art results across both benchmarks. On PASCAL-$5^i$, using a ResNet-50 backbone, the model reached a mean Intersection over Union (mIoU) of 54.03% in the 1-shot setting and 64.04% in the 5-shot setting.

Quantitative comparison of PAIA against state-of-the-art methods on the PASCAL-5i dataset across different splits.

Key findings include:

  • Efficiency: PAIA outperformed the PPNet baseline by over 1% mIoU while requiring fewer auxiliary images (2 vs. 3).
  • Robustness: The framework remains effective even when support masks are replaced with extremely weak annotations like scribbles or bounding boxes, showing improvements of up to 8.2% over baselines.
  • Noise Suppression: The DSMP module proved critical in preventing "background noise" (e.g., a car in the background of a "dog" auxiliary image) from corrupting the class prototype.

Conclusion

PAIA demonstrates that image-level weak annotations are a powerful resource for few-shot semantic segmentation when paired with effective distillation and fusion mechanisms. By dynamically filtering noise and iteratively refining prototypes, PAIA sets a new standard for performance and annotation efficiency in the field. Future research could explore the application of these statistical filtering techniques to fully unsupervised or panoptic segmentation tasks.

Summary updated

Abstract

Despite the great progress made by deep neural networks in the semantic segmentation task, traditional neural-network-based methods typically suffer from a shortage of large amounts of pixel-level annotations. Recent progress in few-shot semantic segmentation tackles the issue by only a few pixel-level annotated examples. However, these few-shot approaches cannot easily be applied to multi-way or weak an-notation settings. In this paper, we advance the few-shot segmentation paradigm towards a scenario where image-level an-notations are available to help the training process of a few pixel-level annotations. Our key idea is to learn a better prototype representation of the class by fusing the knowledge from the image-level labeled data. Specifically, we propose a new framework, called PAIA, to learn the class prototype representation in a metric space by integrating image-level annotations. Furthermore, by considering the uncertainty of pseudo-masks, a distilled soft masked average pooling strategy is designed to handle distractions in image-level annotations. Extensive empirical results on two datasets show superior performance of PAIA.
BibTeX
@inproceedings{lei2021few,
  author = {Shuo Lei and Xuchao Zhang and Jianfeng He and Fanglan Chen and Chang-Tien Lu},
  title = {{Few-Shot Semantic Segmentation Augmented with Image-Level Weak Annotations}},
  booktitle = {ICME},
  year = {2021},
  url = {https://openreview.net/forum?id=Y3anhRiNHb2},
}
Figure