Exploring High-quality Target Domain Information for Unsupervised Domain Adaptive Semantic Segmentation

Junjie Li, Zilei Wang, Yuan Gao, Xiaoming Hu

Abstract

In unsupervised domain adaptive (UDA) semantic segmentation, distillation-based methods currently dominate performance. However, these techniques require complex multi-stage processes and numerous training tricks. This paper proposes a simple yet effective method that achieves competitive performance compared to advanced distillation methods. Our core idea is to fully explore target-domain information through boundaries and features. First, we introduce a novel mix-up strategy to generate high-quality target-domain boundaries with ground-truth labels. By selecting high-confidence target-domain areas and pasting them onto source-domain images, we generate object boundaries in the target domain with correct labels. Second, we design a multi-level contrastive loss to improve target-domain data representation, including pixel-level and prototype-level contrastive learning. Experimental results on GTA5 → Cityscapes and SYNTHIA → Cityscapes benchmarks show that our method achieves state-of-the-art performance, notably reaching 57.8% mIoU on the SYNTHIA 16-class task.

Introduction

Semantic segmentation—the task of assigning a category (e.g., "car," "road," or "sidewalk") to every pixel in an image—is a cornerstone of computer vision, yet it is notoriously data-hungry. While synthetic environments like GTA5 can generate infinite labeled data, models trained on these "fake" datasets often fail when deployed in real-world scenarios, such as the Cityscapes dataset, due to the domain gap.

To bridge this gap, researchers utilize Unsupervised Domain Adaptation (UDA). Currently, the most successful UDA methods rely on "knowledge distillation," a complex process involving multiple training stages and specialized initialization. While effective, these methods are computationally expensive and difficult to implement. Simpler "non-distillation" methods exist but frequently struggle with object boundaries. In the target domain, models are often confident about the center of an object but fail at the edges. Because pseudo-labels at these edges are unreliable, the model never learns to sharpen those boundaries. This paper introduces EHTDI (Exploring High-quality Target Domain Information), a framework designed to address these boundary issues and feature representation without the overhead of distillation.

Methodology

The EHTDI framework focuses on two novel components to maximize the utility of target-domain data:

1. High-confidence Target domain pixel Cross-domain Mixing (HTCM)

Standard UDA mixing strategies often paste source pixels onto target images. HTCM reverses this logic. It identifies pixels in the target image that the model predicts with high confidence (typically the interior of objects) and pastes them onto source images. This strategy "forces" these high-confidence target pixels to become boundary pixels in a new, mixed context. Because these pixels have reliable pseudo-labels, the model effectively learns target-domain boundary characteristics using high-quality supervision, effectively manufacturing reliable training data for the most difficult parts of the image.

2. Multi-level Contrastive Learning

To make features more discriminative without explicit labels, EHTDI implements two types of contrastive losses:

  • Pixel-level Contrastive Learning (PiCL): This ensures that the same pixel, when viewed through different data augmentations, results in similar feature representations, promoting local consistency.
  • Prototype-level Contrastive Learning (ProCL): This ensures that features of the same category (e.g., all "car" pixels) are pulled toward a central "prototype" while being pushed away from prototypes of other categories, improving global class separation.

Experimental Results

The study evaluated EHTDI on two industry-standard "synthetic-to-real" benchmarks: GTA5 → Cityscapes and SYNTHIA → Cityscapes.

Performance Benchmarks

The proposed method achieved state-of-the-art results for non-distillation methods and even surpassed leading distillation-based models:

  • GTA5 → Cityscapes: EHTDI achieved 58.8% mIoU, outperforming the previous leading distillation method, ProDA, by 1.3%.
  • SYNTHIA → Cityscapes: The method achieved 57.8% mIoU (on 16 classes) and 64.6% mIoU (on 13 classes), representing a significant 4.5% lead over the next best non-distillation method.

Efficiency Gains

Beyond accuracy, EHTDI offers a massive leap in training efficiency. While complex models like ProDA require approximately 108 hours of training on four high-end GPUs, EHTDI requires only 63 hours on a single RTX 3090Ti. This reduction in hardware requirements and time makes high-performance UDA much more accessible.

Discussion and Insights

The success of EHTDI stems from a fundamental realization regarding boundaries: while pseudo-labels for target boundaries are inherently noisy, the interior pixels are usually correct. By repositioning those interior pixels to the edges via cross-domain mixing, the model can learn the specific textures and characteristics of the target domain without being misled by incorrect labels.

Furthermore, by performing contrastive learning on mixed images rather than just the target images, the model learns features that are invariant across both the source and target domains. This proves that a well-designed single-stage training process can outperform "heavy," multi-stage distillation models.

Conclusion

EHTDI demonstrates that exploring high-quality target domain information through strategic mixing and multi-level contrastive learning is a highly effective alternative to complex distillation techniques. By achieving state-of-the-art results with significantly lower computational costs, this research suggests that the "non-distillation" philosophy is a viable and efficient route for the future of real-time UDA systems.

Summary updated

Abstract

In unsupervised domain adaptive (UDA) semantic segmentation, the distillation based methods are currently dominant in performance. However, the distillation technique requires complicate multi-stage process and many training tricks. In this paper, we propose a simple yet effective method that can achieve competitive performance to the advanced distillation methods. Our core idea is to fully explore the target-domain information from the views of boundaries and features. First, we propose a novel mix-up strategy to generate high-quality target-domain boundaries with ground-truth labels. Different from the source-domain boundaries in previous works, we select the high-confidence target-domain areas and then paste them to the source-domain images. Such a strategy can generate the object boundaries in target domain (edge of target-domain object areas) with the correct labels. Consequently, the boundary information of target domain can be effectively captured by learning on the mixed-up samples. Second, we design a multi-level contrastive loss to improve the representation of target-domain data, including pixel-level and prototype-level contrastive learning. By combining two proposed methods, more discriminative features can be extracted and hard object boundaries can be better addressed for the target domain. The experimental results on two commonly adopted benchmarks (\textit{i.e.}, GTA5 $\rightarrow$ Cityscapes and SYNTHIA $\rightarrow$ Cityscapes) show that our method achieves competitive performance to complicated distillation methods. Notably, for the SYNTHIA$\rightarrow$ Cityscapes scenario, our method achieves the state-of-the-art performance with $57.8\%$ mIoU and $64.6\%$ mIoU on 16 classes and 13 classes. Code is available at https://github.com/ljjcoder/EHTDI.
BibTeX
@inproceedings{li2022exploring,
  author = {Junjie Li and Zilei Wang and Yuan Gao and Xiaoming Hu},
  title = {{Exploring High-quality Target Domain Information for Unsupervised Domain Adaptive Semantic Segmentation}},
  booktitle = {ACM Multimedia},
  year = {2022},
  url = {https://openreview.net/forum?id=vTH6Lc4mn5},
}
Figure