Gated CRF Loss for Weakly Supervised Semantic Image Segmentation

Luc Van GoolarXiv 2019 · arXiv

TL;DR

This paper introduces a novel "Gated CRF Loss" to improve weakly supervised semantic image segmentation. By intelligently combining a standard cross-entropy loss for labeled pixels with a refined Conditional Random Field (CRF) loss for unlabeled pixels, the method significantly bridges the performance gap between weakly and fully supervised approaches, achieving state-of-the-art results on click and scribble-based annotations.

What Problem Does This Paper Address?

Semantic image segmentation, a crucial task for many computer vision applications, typically relies on deep neural networks trained with pixel-perfect annotations. However, creating these detailed labels is extremely time-consuming and expensive. Weakly supervised methods aim to reduce this annotation burden by using less precise forms of supervision like scribbles or clicks. While these methods are more efficient, they often struggle to produce accurate object boundaries, leading to a significant performance drop compared to fully supervised methods. Existing weakly supervised techniques often employ complex, task-specific modules or iterative training schemes, hindering reusability and steady progress. This paper seeks to address this by proposing a simple yet effective loss function that can be easily integrated into standard training pipelines.

Methods

The core contribution is the Gated CRF Loss. The authors propose training standard neural networks by jointly minimizing two components:

  1. Partial Cross-Entropy (pCE) Loss: This is the standard cross-entropy loss applied only to the pixels that have been weakly annotated (e.g., clicks or scribbles).
  2. Gated CRF Loss: This novel loss component is applied to the unlabeled pixels. It's based on the principles of Conditional Random Fields (CRFs), which model relationships between neighboring pixels to refine segmentation boundaries. The "gating" mechanism is key:
  3. Source Masking: It allows the model to selectively ignore contributions from certain pixels when calculating pairwise energies, preventing noisy or irrelevant pixels from negatively impacting the learning process.
  4. Destination Masking: It controls which pixels can receive and accumulate pairwise energy, allowing the model to focus learning on specific regions.
  5. Local Neighborhood: Instead of considering all pixel pairs (which is computationally expensive), the Gated CRF loss restricts the pairwise interactions to a local neighborhood around each pixel. This is justified by the observation that modern CNNs already capture global context effectively.

This approach allows the Gated CRF loss to focus on learning contextual relationships and refining semantic boundaries without relying on complex, high-dimensional filtering, making it amenable to efficient GPU implementation and seamless integration with deep learning frameworks.

Datasets

The proposed method was evaluated on two widely used datasets:

  • Pascal VOC 2012: This dataset contains images with dense semantic annotations for 21 classes. The authors used publicly available scribble annotations from ScribbleSup and click annotations from the "What's the point" paper.
  • Cityscapes: This dataset features urban driving scenes with dense semantic annotations for 19 classes. Since no public weak annotations were available, the authors simulated click annotations from the dense annotations.

Key Insights and Ideas

  • The "Purist" Approach: The paper advocates for a simple, integrated approach that avoids complex pre- or post-processing steps, aiming for a more generalizable solution.
  • Gating is Crucial: The introduction of "source" and "destination" masks in the CRF loss is shown to be vital for improving performance, especially when dealing with sparse weak annotations.
  • Local Neighborhood Suffices: The authors demonstrate that restricting CRF interactions to a local neighborhood is sufficient for achieving good results, likely due to the inherent global context aggregation capabilities of modern CNN architectures.
  • Bridging the Gap: The Gated CRF loss effectively reduces the performance gap between weakly and fully supervised semantic segmentation, bringing the former closer to the latter.

Key Results

The Gated CRF loss achieved state-of-the-art performance on the Pascal VOC dataset for both click and scribble-based weak supervision:

  • Click Supervision (Pascal VOC): Achieved an mIoU of 67.39%, significantly outperforming previous methods (e.g., 57.00% from a previous SOTA).
  • Scribble Supervision (Pascal VOC): Achieved an mIoU of 75.50%, also outperforming previous methods (e.g., 72.90% from a previous SOTA).
  • The method also showed strong performance on the Cityscapes dataset.
  • The paper highlights that their method with click supervision is on par with bounding-box supervised approaches, despite clicks being a much cheaper annotation.
  • The performance gap between weak and full supervision was significantly narrowed, reaching up to 87.88% for clicks and 98.46% for scribbles in terms of the Weak/Full ratio.

Limitations and Future Work

  • Dataset-Specific Masking: The effectiveness of destination masking was found to be dataset-dependent, suggesting that its application might require careful consideration for different datasets.
  • Hyperparameter Tuning: While the Gated CRF loss is shown to be less prone to loss balancing issues, the optimal weight ($\lambda$) for the CRF loss still needs to be determined.
  • Simplified Pairwise Term: The paper mentions that the pairwise energy term could be further simplified and potentially implemented using similar high-dimensional filtering techniques, but this is left as future work.
  • Broader Weak Supervision: While the paper focuses on clicks and scribbles, the authors suggest that the proposed techniques could be extended to other forms of weak supervision, like image-level tags.

Summary updated

Abstract

State-of-the-art approaches for semantic segmentation rely on deep convolutional neural networks trained on fully annotated datasets, that have been shown to be notoriously expensive to collect, both in terms of time and money. To remedy this situation, weakly supervised methods leverage other forms of supervision that require substantially less annotation effort, but they typically present an inability to predict precise object boundaries due to approximate nature of the supervisory signals in those regions. While great progress has been made in improving the performance, many of these weakly supervised methods are highly tailored to their own specific settings. This raises challenges in reusing algorithms and making steady progress. In this paper, we intentionally avoid such practices when tackling weakly supervised semantic segmentation. In particular, we train standard neural networks with partial cross-entropy loss function for the labeled pixels and our proposed Gated CRF loss for the unlabeled pixels. The Gated CRF loss is designed to deliver several important assets: 1) it enables flexibility in the kernel construction to mask out influence from undesired pixel positions; 2) it offloads learning contextual relations to CNN and concentrates on semantic boundaries; 3) it does not rely on high-dimensional filtering and thus has a simple implementation. Throughout the paper we present the advantages of the loss function, analyze several aspects of weakly supervised training, and show that our `purist' approach achieves state-of-the-art performance for both click-based and scribble-based annotations.
BibTeX
@misc{gool2019gated,
  author = {Luc Van Gool},
  title = {{Gated CRF Loss for Weakly Supervised Semantic Image Segmentation}},
  year = {2019},
  eprint = {1906.04651},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1906.04651},
}
Figure