Single Stage Weakly Supervised Semantic Segmentation of Complex Scenes

Peri Akiva, Kristin DanaWACV 2023

TL;DR

This paper introduces a novel single-stage, weakly supervised semantic segmentation (WSSS) method that can be trained from scratch without pre-trained backbones. It addresses the limitations of existing WSSS methods that struggle with complex scenes by using point annotations to generate reliable pseudo-masks, achieving state-of-the-art performance on challenging real-world datasets.

What Problem Does This Paper Address?

Semantic segmentation, the task of assigning a class label to every pixel in an image, is crucial for many computer vision applications. However, obtaining pixel-level annotations is extremely time-consuming and expensive. Weakly supervised semantic segmentation (WSSS) aims to overcome this by using cheaper annotations like image-level labels, points, or bounding boxes.

Existing WSSS methods often fall into two camps: those using image-level labels, which rely heavily on class activation maps (CAMs) and struggle with complex scenes (high object count, scale variation, or diversity), and those using point or box annotations, which often make strong assumptions about object scale or dataset diversity. This leads to methods that are overly specialized and perform poorly on datasets outside their specific complexity range. Furthermore, many WSSS methods require multi-stage training, making them less practical for real-world applications. This paper tackles the challenge of developing a WSSS method that is generalizable to a wide range of scene complexities, trainable from scratch, and efficient enough for practical use.

Methods

The proposed method, dubbed "Pseudo-Masks from Points" (PMP), is a single-stage approach that leverages point annotations to generate high-quality pseudo-masks for training a semantic segmentation network. Key novel components include:

  • Point Blot Generator: This module takes sparse point annotations and iteratively generates a denser set of "point blots" by applying random affine transformations and random walks. This captures more local contextual information around the initial points, providing a richer seed for pseudo-mask generation.
  • Expanding Distance Fields: This module uses the point annotations to create distance fields that encode spatial information. An "expansion mechanism" adaptively adjusts the influence of these fields based on the network's learning progress, allowing for more robust localization and preventing the accumulation of errors in early training stages, especially for large objects.
  • Pixel Adaptive Convolution (PAC) Refinement Network: This network refines the features generated by the backbone. It uses PAC layers to dynamically adjust kernel weights based on local color and feature statistics, dampening inconsistent regions and amplifying locally similar ones. This helps to smooth and ensure spatial accuracy of features.

These components work together: the PAC Refiner smooths features, the Expanding Distance Fields provide spatial guidance, and the Point Blot Generator enriches the initial supervision. The final pseudo-mask is generated by combining these refined features and distance fields, which then supervises the segmentation network trained from scratch.

Datasets

The method was trained and evaluated on six datasets:

  • Pascal VOC 2012: A standard benchmark dataset.
  • Cranberry from Aerial Imagery Dataset (CRAID): A real-world dataset with a high object count.
  • CityPersons: A real-world dataset with a high object count.
  • Inria Aerial Dataset (IAD): A real-world dataset with varying object scales.
  • ADE20K: A real-world dataset with high scene diversity.
  • CityScapes: A real-world dataset with high scene diversity.

The paper highlights that these real-world datasets represent a wider range of complexities than typically explored by image-level WSSS methods.

Key Insights and Ideas

  • Point annotations are a sweet spot: They offer sufficient spatial information for localization without the extreme cost of pixel-level annotations, and are more robust than image-level labels for complex scenes.
  • Single-stage training is achievable and beneficial: By carefully designing the pseudo-mask generation process, the authors demonstrate that a single-stage approach can achieve competitive results without the complexity and inflexibility of multi-stage methods.
  • Iterative refinement and expansion are crucial: The combination of PAC refinement for local feature smoothing and Expanding Distance Fields for global spatial consistency, coupled with an adaptive expansion mechanism, is key to generating reliable pseudo-masks from sparse points.
  • Generalizability is paramount: The paper emphasizes the limitations of methods tuned to specific dataset complexities and aims to provide a solution that works across a broad spectrum of scenes.

Key Results

The proposed method achieves state-of-the-art (SOTA) performance on several challenging real-world datasets:

  • Significant improvements over baselines: The method outperforms existing single-stage WSSS baselines by up to 28.1% and multi-stage baselines by up to 22.6% on datasets like CRAID, CityPersons, IAD, ADE20K, and CityScapes.
  • Competitive performance on Pascal VOC 2012: It achieves competitive results on this standard benchmark.
  • Trainable from scratch: Notably, the method achieves these results without relying on pre-trained backbones, which is a significant advantage for adapting to new or specialized datasets.
  • Qualitative results: The generated pseudo-masks are shown to converge towards fully-supervised ground truth, producing more refined predictions than baselines, especially on complex scenes.

Limitations and Future Work

  • Limitations: The paper acknowledges that while point annotations are cheaper than pixel-level ones, they still require some annotation effort. The method's performance is dependent on the quality and distribution of these initial points.
  • Future Work: The authors suggest that their method's generalizability and single-stage nature make it suitable for application-driven systems. Further exploration of different types of weak supervision and their integration could be a future direction. The paper also mentions that ablation studies and additional qualitative results are available in the supplementary material, implying potential for further analysis.

Summary updated

BibTeX
@inproceedings{akiva2023single,
  author = {Peri Akiva and Kristin Dana},
  title = {{Single Stage Weakly Supervised Semantic Segmentation of Complex Scenes}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure