Uncertainty-Aware Semi-Supervised Few Shot Segmentation

Soopil Kim, Philip Chikontwe, Sang Hyun ParkarXiv 2021 · arXiv

TL;DR

This paper introduces an uncertainty-aware semi-supervised framework for Few-Shot Segmentation (FSS). By estimating the reliability of "pseudo-labels" on unlabeled images, the model filters out noisy predictions to generate high-quality additional prototypes, significantly boosting segmentation accuracy on novel classes with minimal labeled data.

What Problem Does This Paper Address?

Few-Shot Segmentation (FSS) is the task of segmenting a specific object in a "query" image given only one or a few labeled "support" images. The core challenge is the distributional shift: the model must generalize to object classes it has never seen during training.

Because support sets are tiny (often just 1 image), they rarely capture the full variety of an object's appearance. Semi-supervised FSS attempts to fix this by using unlabeled images to find more examples of the target object. However, this relies on "pseudo-labels"—the model's own best guess at where the object is in the unlabeled data. If these guesses are wrong or overconfident, they "poison" the model with bad information, leading to poor performance.

Methods

The authors build upon a prototype-based approach (specifically PPNet), where objects are represented by "prototypes" (feature vectors) in a high-dimensional space. Their key innovations include:

  1. Neural Uncertainty Estimation: Instead of just predicting a class probability ($\mu$), the model is trained to estimate the variance ($\sigma^2$) of its predictions. Inspired by Conditional Neural Processes, they treat pixel-level classification as a Gaussian distribution and optimize it using a Negative Log-Likelihood (NLL) loss.
  2. Uncertainty-Guided Refinement: They define a refined probability $\mu' = \mu \times (1 - \sigma)$. This formula ensures that even if a pixel has a high initial probability, it will be excluded from the pseudo-label if the model is uncertain about it.
  3. Semi-Supervised Prototype Generation: The model uses these "cleaned" pseudo-labels on unlabeled images to extract additional prototypes ($P_u$). These complement the original support prototypes ($P_s$), providing a richer representation of the target object's parts and variations.
  4. Refinement Module ($R$): A final stage that uses low-level features and an Atrous Spatial Pyramid Pooling (ASPP) module to sharpen object boundaries, which are often blurry in standard FSS outputs.
Figure 2: Overview of the proposed semi-supervised FSS inference strategy. Blue box shows a detail of refinement module.

Datasets

The model was evaluated on two standard FSS benchmarks:

  • PASCAL-5$^i$: A subset of PASCAL VOC 2012 containing 20 classes.
  • COCO-20$^i$: A more challenging dataset derived from MS COCO, featuring 80 classes. The authors used ResNet-50 and ResNet-101 as backbones for their experiments.

Key Insights and Ideas

  • Uncertainty as a Filter: The "aha moment" is that you don't need perfect pseudo-labels to benefit from unlabeled data; you just need to know which parts of the pseudo-label to trust. By discarding high-uncertainty regions, the model avoids learning from its own mistakes.
  • Gaussian Modeling vs. Sampling: Traditional uncertainty methods (like Monte Carlo Dropout) require multiple passes through the network, which is slow. This paper's Gaussian approach allows for single-pass uncertainty estimation, making it much more efficient for real-time inference.
  • Filling the Feature Space: The t-SNE visualization (Figure 4 in the paper) reveals that unlabeled prototypes ($P_u$) often land in regions of the feature space that the limited support set ($P_s$) missed, effectively "filling the gaps" and allowing the model to recognize more parts of the query object.
Figure 4: A t-SNE visualization of query features and proto- types from support and unlabeled images.

Key Results

The proposed model achieved state-of-the-art (SOTA) results on both benchmarks, particularly in the 5-shot setting.

  • PASCAL-5$^i$: Using ResNet-101 in a 5-shot setting, the model reached 67.2% mean-IoU, outperforming the previous baseline (PPNet) by a significant margin.
  • COCO-20$^i$: The model achieved 43.9% mean-IoU in the 5-shot setting, setting a new high bar for this difficult dataset.
  • Qualitative Improvement: As shown in the visualization below, the uncertainty map ($\sigma_{fg}$) correctly identifies ambiguous regions (like boundaries or similar-looking background objects), leading to much cleaner pseudo-labels ($PL'$) compared to the raw predictions ($PL$).
Visualization of intermediate results showing how uncertainty refines pseudo-labels.

Limitations and Future Work

  • Baseline Dependency: The authors acknowledge that the 1-shot performance is somewhat constrained by the strength of the underlying baseline model. If the initial guess is extremely poor, uncertainty estimation can only do so much.
  • Computational Overhead: While more efficient than sampling-based methods, the addition of a refinement module and the processing of multiple unlabeled images does increase the computational cost compared to simple supervised FSS.
  • Future Directions: The authors suggest that this uncertainty-aware concept is generalizable and could be integrated into other prototype-based FSS architectures to further push the boundaries of the field.

Summary updated

Abstract

Few shot segmentation (FSS) aims to learn pixel-level classification of a target object in a query image using only a few annotated support samples. This is challenging as it requires modeling appearance variations of target objects and the diverse visual cues between query and support images with limited information. To address this problem, we propose a semi-supervised FSS strategy that leverages additional prototypes from unlabeled images with uncertainty guided pseudo label refinement. To obtain reliable prototypes from unlabeled images, we meta-train a neural network to jointly predict segmentation and estimate the uncertainty of predictions. We employ the uncertainty estimates to exclude predictions with high degrees of uncertainty for pseudo label construction to obtain additional prototypes based on the refined pseudo labels. During inference, query segmentation is predicted using prototypes from both support and unlabeled images including low-level features of the query images. Our approach is end-to-end and can easily supplement existing approaches without the requirement of additional training to employ unlabeled samples. Extensive experiments on PASCAL-$5^i$ and COCO-$20^i$ demonstrate that our model can effectively remove unreliable predictions to refine pseudo labels and significantly improve upon state-of-the-art performances.
BibTeX
@misc{kim2021uncertainty,
  author = {Soopil Kim and Philip Chikontwe and Sang Hyun Park},
  title = {{Uncertainty-Aware Semi-Supervised Few Shot Segmentation}},
  year = {2021},
  eprint = {2110.08954},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2110.08954},
}

Loading related content...

Figure