Edge-aware Plug-and-play Scheme for Semantic Segmentation

Jianye Yi, Xiaopin Zhong, Weixiang Liu, Wenxuan Zhu, Zongze Wu, Yuanlong DengarXiv 2023 · arXiv · OpenReview

TL;DR

This paper introduces an "Edge-aware Plug-and-play Scheme" (EPS) that allows for easy integration of edge supervision into any semantic segmentation model, improving accuracy without modifying the model's architecture. They also propose a novel "Polar Hausdorff Loss" (PH Loss) to effectively leverage edge information and preserve edge thickness during training.

What Problem Does This Paper Address?

Semantic segmentation, the task of assigning a class label to every pixel in an image, has seen significant advancements with deep learning. However, many state-of-the-art methods are highly specialized and tied to specific network architectures, making them difficult to adapt to new models or tasks. While incorporating edge information has shown promise in improving segmentation accuracy, existing edge supervision methods are often complex, require custom network designs, and are not universally applicable. Furthermore, traditional loss functions used for edge supervision lack the ability to preserve the spatial-geometric characteristics of edges, leading to uncertain boundary predictions. This paper aims to bridge this gap by providing a universal, easy-to-use edge supervision scheme and a novel loss function that addresses these limitations.

Methods

The core contribution is the Edge-aware Plug-and-play Scheme (EPS). This scheme is designed to be architecture-agnostic and can be applied to any semantic segmentation model. It works in two main steps:

  1. Edge Ground Truth (Edge GT) Generation: EPS first generates an "Edge GT" from the original ground truth masks. This Edge GT is created with a predefined edge thickness, denoted as $d_e$. This thickness is controlled by a kernel size ($n \times n$, where $n = 2d_e + 1$).
  2. Auxiliary Decoder Head: For any given semantic segmentation network, EPS directly copies its decoder head. This copied head is then used as an auxiliary task, supervised by the generated Edge GT. Crucially, this auxiliary head's weights are not shared with the original decoder.

To effectively utilize the edge information and preserve the desired edge thickness, the authors propose a novel loss function called Polar Hausdorff (PH) Loss. Unlike traditional Hausdorff Loss, which measures the distance between predicted and ground truth boundaries, PH Loss focuses on the distance between the internal and external edges of the predicted edge segmentation. It aims to constrain this distance to match the predefined edge thickness $d_e$. The PH Loss calculation involves converting pixel coordinates to polar coordinates and using rays to efficiently identify inner and outer edge points, making it computationally feasible.

Datasets

The experiments were conducted on the Cityscapes dataset, a widely used benchmark for semantic segmentation, featuring 5000 high-resolution images with 19 categories.

Key Insights and Ideas

  • Universality of Edge Supervision: The paper demonstrates that edge supervision can be a powerful tool for improving semantic segmentation, and importantly, it can be applied universally across different network architectures (CNNs and ViTs) without requiring architectural modifications.
  • The Importance of Edge Thickness: Explicitly defining and preserving edge thickness during supervision is crucial for accurate boundary segmentation. Traditional methods often overlook this aspect.
  • PH Loss as a Specialized Tool: The proposed PH Loss is specifically designed to leverage the concept of edge thickness, offering a more targeted and effective way to supervise edge prediction compared to general boundary losses.
  • Plug-and-Play Simplicity: The EPS scheme's "plug-and-play" nature is a significant advantage, allowing researchers and practitioners to easily experiment with edge supervision without deep architectural changes.

Key Results

  • The proposed EPS scheme, when combined with the PH Loss, consistently improved the performance of 22 different semantic segmentation models on the Cityscapes dataset.
  • Improvements were observed in both mean Intersection over Union (mIoU) and mean Accuracy (mAcc) metrics. For instance, CGNet saw an increase from 66.84% mIoU to 68.68% with EPS and further to 70.16% with EPS + PH Loss.
  • The paper highlights that EPS can be seamlessly integrated into state-of-the-art (SOTA) models with zero modification, leading to promising enhancements.
  • Ablation studies showed that while the specific kernel size for Edge GT generation ($d_e$) can influence results, the overall approach is robust. The hyperparameter $n$ for PH Loss was found to have minimal impact on performance, suggesting a degree of robustness.

Limitations and Future Work

  • Hyperparameter Sensitivity: While the paper explores hyperparameters like $d_e$ and $n$, the authors acknowledge that the optimal kernel size can vary across models, and further analysis is needed.
  • Robustness Analysis: The paper notes a lack of stability and robustness analysis for the PH Loss.
  • Limited Dataset Scope: Experiments were primarily conducted on the Cityscapes dataset.
  • Future Directions: The authors suggest exploring additional plug-and-play schemes (e.g., texture supervision), optimizing PH Loss calculation, extending EPS to other tasks like instance segmentation and object detection, and applying it to diverse fields such as medical imaging and autonomous driving.

Summary updated

Abstract

Semantic segmentation is a classic and fundamental computer vision problem dedicated to assigning each pixel with its corresponding class. Some recent methods introduce edge-based information for improving the segmentation performance. However these methods are specific and limited to certain network architectures, and they can not be transferred to other models or tasks. Therefore, we propose an abstract and universal edge supervision method called Edge-aware Plug-and-play Scheme (EPS), which can be easily and quickly applied to any semantic segmentation models. The core is edge-width/thickness preserving guided for semantic segmentation. The EPS first extracts the Edge Ground Truth (Edge GT) with a predefined edge thickness from the training data; and then for any network architecture, it directly copies the decoder head for the auxiliary task with the Edge GT supervision. To ensure the edge thickness preserving consistantly, we design a new boundarybased loss, called Polar Hausdorff (PH) Loss, for the auxiliary supervision. We verify the effectiveness of our EPS on the Cityscapes dataset using 22 models. The experimental results indicate that the proposed method can be seamlessly integrated into any state-of-the-art (SOTA) models with zero modification, resulting in promising enhancement of the segmentation performance.
BibTeX
@misc{yi2023edge,
  author = {Jianye Yi and Xiaopin Zhong and Weixiang Liu and Wenxuan Zhu and Zongze Wu and Yuanlong Deng},
  title = {{Edge-aware Plug-and-play Scheme for Semantic Segmentation}},
  year = {2023},
  eprint = {2303.10307},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.10307},
}
Figure