MaskSplit: Self-supervised Meta-learning for Few-shot Semantic Segmentation

Mustafa Sercan Amac, Ahmet Sencan, Orhun Bugra Baran, Nazli Ikizler-Cinbis, Ramazan Gokberk Cinbis

TL;DR

This paper introduces MaskSplit, a novel self-supervised meta-learning framework for few-shot semantic segmentation. By leveraging unsupervised saliency estimation and clever image splitting techniques, it eliminates the need for manual annotations during training, significantly reducing data requirements for segmenting new object classes.

What Problem Does This Paper Address?

Semantic segmentation, the task of assigning a label to every pixel in an image, has seen tremendous progress. However, state-of-the-art methods typically require vast amounts of meticulously annotated training data, making them expensive and time-consuming to develop. Few-shot semantic segmentation (FSS) aims to alleviate this by enabling models to segment novel object classes with only a few annotated examples (support set). Despite this reduction in annotation needs for new classes, FSS models still rely on large, annotated datasets of base classes for meta-learning or pre-training. This paper tackles the fundamental challenge of reducing annotation dependency even further by proposing a self-supervised approach to learn the FSS meta-model itself, entirely from unlabeled data.

Methods

The core of the proposed method, MaskSplit, lies in its innovative approach to generating training data without manual labels. Here's a breakdown:

  • Unsupervised Saliency Estimation: The process begins by using an unsupervised saliency model (trained on a separate dataset) to generate "pseudo-masks" for images. These masks highlight salient regions, acting as a proxy for ground truth.
  • MaskSplitting: To create the support and query pairs needed for meta-learning, the pseudo-masks are split. The authors propose several variants:
  • Vsplit: Splits the mask vertically.
  • Hsplit: Splits the mask horizontally.
  • MaskSplit with Slope: Introduces a slight tilt to the splitting line to avoid axis-aligned biases.
  • MaskSplit with Alternating Sides: Randomly assigns one split half as the support and the other as the query in each episode.
  • Augmentations: To ensure the support and query pairs are sufficiently different and to improve generalization, standard image augmentations (like color jitter, flips, rotations) are applied.
  • Prototype-Based Network: The meta-model itself is a prototype-based network, inspired by existing FSS architectures. It consists of a backbone (ResNet-101) to extract features, a mixing module to combine support and query features, and an upsampling module to predict the segmentation mask.
  • Loss Evaluation: A crucial detail is that the loss is not calculated on query pixels corresponding to the support area to avoid negative predictions.

Datasets

The paper evaluates its approach on two standard FSS benchmarks:

  • PASCAL-5²: Derived from PASCAL VOC 2012, this dataset is divided into four folds. The authors utilize all folds for unsupervised training, deviating from traditional fold-specific training.
  • COCO-20i: Based on the COCO dataset, this benchmark has 80 classes divided into four folds. Similar to PASCAL-5², the model is trained unsupervisedly on all folds.

Key Insights and Ideas

  • Saliency as a Proxy for Ground Truth: The paper demonstrates that unsupervised saliency maps, while imperfect, can serve as a viable substitute for manual annotations in the context of self-supervised FSS training.
  • MaskSplitting for Episodic Training: The core innovation of splitting saliency masks to create artificial support and query pairs is a clever way to generate diverse training episodes without any human labels. This is the first work to apply this to unsupervised FSS.
  • Augmentations are Crucial: The experiments highlight that while MaskSplit is effective, combining it with strong image augmentations significantly boosts performance, emphasizing the need for visually distinct support and query examples.
  • Self-Supervised Outperforms Supervised (in some settings): Interestingly, the self-supervised approach trained on the entire dataset (all folds) achieved comparable or even better results than a supervised baseline trained on specific folds, suggesting the benefits of leveraging more data.

Key Results

  • Competitive Performance: The proposed MaskSplit framework achieves competitive results on the PASCAL-5² dataset, outperforming unsupervised baselines by a significant margin (e.g., ~4% mIoU over MaskContrast).
  • Ablation Study Highlights: Ablation studies show that augmentations are crucial, contributing to a substantial performance increase. The "slope" component in MaskSplit also offers a notable improvement.
  • Oracle Performance: When using ground truth masks instead of saliency, the model achieves an impressive 57.3% mIoU on PASCAL-5², indicating the potential of the underlying architecture and training paradigm.
  • COCO-20i Results: On COCO-20i, the model also outperforms the saliency baseline, though the performance gap is smaller, possibly due to noisier saliency maps on this dataset.

Limitations and Future Work

  • Saliency Map Imperfections: The reliance on unsupervised saliency maps means that the quality of these maps directly impacts performance. Noisy saliency can lead to suboptimal training.
  • Dataset Specificity: While evaluated on PASCAL and COCO, the performance on other datasets, especially those with very different visual characteristics, is not explored.
  • Future Work: The authors suggest that their work stimulates further research into unsupervised learning for few-shot segmentation. Potential future directions could involve exploring more advanced unsupervised saliency methods or alternative self-supervised pretext tasks tailored for FSS. The paper also notes that the performance gap against supervised methods, while shrinking, still exists, suggesting room for improvement.

Summary updated

Abstract

Just like other few-shot learning problems, few-shot segmentation aims to minimize the need for manual annotation, which is particularly costly in segmentation tasks. Even though the few-shot setting reduces this cost for novel test classes, there is still a need to annotate the training data. To alleviate this need, we propose a self-supervised training approach for learning few-shot segmentation models. We first use unsupervised saliency estimation to obtain pseudo-masks on images. We then train a simple prototype based model over different splits of pseudo masks and augmentations of images. Our extensive experiments show that the proposed approach achieves promising results, highlighting the potential of self-supervised training. To the best of our knowledge this is the first work that addresses unsupervised few-shot segmentation problem on natural images.
BibTeX
@inproceedings{amac2022masksplit,
  author = {Mustafa Sercan Amac and Ahmet Sencan and Orhun Bugra Baran and Nazli Ikizler-Cinbis and Ramazan Gokberk Cinbis},
  title = {{MaskSplit: Self-supervised Meta-learning for Few-shot Semantic Segmentation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
  url = {https://openreview.net/forum?id=OwQ3zVpBkxc},
}
Figure