Robust unsupervised domain adaptation by retaining confident entropy via edge concatenation

Abhishek Kumar, Hye-Seong Hong, Abhishek Kumar 0010, Dong-Gyu LeeExpert Systems with Applications 2024 · DOI

TL;DR

This paper tackles the challenge of unsupervised domain adaptation (UDA) for semantic segmentation by proposing a novel method that integrates both internal (semantic, depth) and external (edge) information. By concatenating edge probabilities with entropy-based predictions, their approach significantly improves the accuracy of object boundary delineation, leading to state-of-the-art performance on challenging synthetic-to-real benchmarks.

What Problem Does This Paper Address?

Semantic segmentation, the task of assigning a class label to every pixel in an image, is crucial for understanding visual scenes. Training these models typically requires vast amounts of pixel-level annotated data, which is expensive and time-consuming to acquire. Unsupervised Domain Adaptation (UDA) aims to bridge this gap by training models on a labeled "source" domain (often synthetic data) and adapting them to an unlabeled "target" domain (real-world data) with different distributions.

Existing UDA methods often focus on aligning feature distributions between domains, but they tend to overlook the importance of explicit object boundaries. While some methods leverage entropy to measure uncertainty and guide adaptation, this entropy can be high even for non-edge regions, leading to confusion. The paper identifies a critical gap: the precise delineation of object boundaries, which is vital for robust semantic segmentation, is often neglected in UDA. They propose to address this by incorporating edge information, which humans naturally use to perceive objects, into the UDA process.

Methods

The core of the proposed method lies in enriching the discriminator of an adversarial UDA framework with edge information. Here's a breakdown of their approach:

  • Feature Extraction: A standard encoder-decoder network (DeepLabv2 with ResNet-101 backbone) is used to extract semantic and depth features from input images.
  • Edge Label Generation: Crucially, edge labels are not obtained from external annotation. Instead, they are generated by applying Canny edge detection to the ground truth semantic segmentation maps of the source domain. This provides a cost-effective way to acquire edge information.
  • Entropy Calculation: Entropy maps are computed from the predicted semantic probability distributions. This captures the uncertainty in the model's predictions for internal regions.
  • Edge Probability Prediction: A separate network predicts edge probabilities from the extracted features. These are then normalized using a sigmoid function.
  • Concatenation and Fusion: The key innovation is the fusion of these different information sources. The predicted edge probabilities are concatenated with the semantic and depth entropy maps. This combined representation is fed to the discriminator. The authors explore different fusion strategies, finding that simple concatenation of edge probabilities with the entropy maps (semantic + depth + edge) yields the best results.
  • Adversarial Training: The discriminator is trained to distinguish between source and target domain features. The segmentation network (generator) is trained to fool the discriminator, thereby aligning the domains.
  • Self-Training: A self-training mechanism is employed, where the model uses its own high-confidence predictions on the target domain as pseudo-labels to further refine its performance.

Datasets

  • Source Domain: SYNTHIA (Ros et al., 2016) - a synthetic dataset with 9,400 RGB images and 12 class categories, including annotated depth labels.
  • Target Domains:
    • Cityscapes (Cordts et al., 2016) - a real-world dataset with 2,495 training and 500 testing images from urban scenes, with 19 classes.
  • Mapillary (Neuhold et al., 2017) - another real-world dataset with 24,966 high-resolution images and 66 object categories.

The paper also evaluates performance on a lower resolution version of these datasets (320x640).

Key Insights and Ideas

  • External information is crucial for UDA: The paper strongly advocates for incorporating external cues like object edges, which are naturally used in human perception, to improve UDA performance, especially for boundary delineation.
  • Cost-effective edge generation: Generating edge labels from existing semantic ground truth (via Canny edge detection) is a smart way to leverage available data without requiring new annotations.
  • Entropy as a proxy for uncertainty, but edges provide definitive cues: While entropy helps identify uncertain regions, it can be high even for non-edge areas. Edges provide more precise, localized information that complements entropy.
  • Concatenation is key for fusion: The authors found that simple concatenation of edge probabilities with semantic and depth information was more effective than other fusion strategies (like element-wise multiplication) for the discriminator.

Key Results

The proposed method achieves state-of-the-art results on several UDA benchmarks:

  • SYNTHIA → Cityscapes (16 classes): The proposed method achieved the highest mIoU (mean Intersection over Union) of 46.10%, outperforming previous methods like CTRL (45.3%) and DADA (43.8%).
  • SYNTHIA → Mapillary (7 classes): The method achieved an mIoU of 69.6%, again surpassing CTRL (68.9%) and DADA (67.6%).
  • Improvements in specific classes: The paper highlights significant improvements in classes like "person" (+2.6%), "vegetation" (+3.2%), and "sky" (+3.2%) on the Cityscapes dataset compared to CTRL.
  • Robustness at boundaries: Qualitative results (Figures 3, 4, and 5) demonstrate that the proposed method produces finer segmentation outputs with better context details near boundaries, showing robustness where previous methods struggled.
  • Low-resolution performance: Even at lower resolutions (320x640), the method maintained competitive performance, showing its adaptability.

Limitations and Future Work

  • Challenges with intricate features: The paper acknowledges limitations in classes with intricate features like fences and lighting conditions. Achieving outstanding performance in these cases is challenging due to the need for highly localized detection and potential confusion with adjacent pixel labels.
  • Future work: The authors suggest exploring techniques for enhancing fine-grained feature detection and strategies to mitigate confusion in highly localized detection scenarios to further improve robustness. They also mention that future work could explore other external cues beyond edges.

Summary updated

BibTeX
@article{kumar2024robust,
  author = {Abhishek Kumar and Hye-Seong Hong and Abhishek Kumar and Dong-Gyu Lee},
  title = {{Robust unsupervised domain adaptation by retaining confident entropy via edge concatenation}},
  journal = {Expert Systems with Applications},
  year = {2024},
  doi = {10.1016/j.eswa.2023.122120},
}
Figure