Domain Adaptive Semantic Segmentation via Entropy-Ranking and Uncertain Learning-Based Self-Training

Chengli Peng, Jiayi MaIEEE CAA J. Autom. Sinica 2022 · OpenReview

Abstract

This letter develops two new self-training strategies for domain adaptive semantic segmentation, which formulate self-training into the processes of mining more training samples and reducing influence of the false pseudo-labels. Particularly, a self-training strategy based on entropy-ranking is proposed to mine intra-domain information. Thus, numerous false pseudo-labels can be exploited and rectified, and more pseudo-labels can be involved in training. Meanwhile, another novel self-training strategy is developed to handle the regions that may possess false pseudo-labels. In detail, a specific uncertain loss, that makes the network automatically decide whether the pseudo-labels are true, is proposed to improve the network optimization. Consequently, the influence of false pseudo-labels can be reduced. Experimental results prove that, compared with the baseline, the average mIoU performance gain brought by our method can attain 4.3%. Extensive benchmark experiments further highlight the effectiveness of our method against existing state-of-the-arts.

Introduction

Semantic segmentation, the task of assigning a class label to every pixel in an image, is a cornerstone of computer vision but typically requires vast amounts of manually annotated data. Unsupervised Domain Adaptation (UDA) offers a solution by training models on labeled synthetic data (source domain) and adapting them to unlabeled real-world data (target domain).

A common technique in UDA is self-training, where a model generates "pseudo-labels" for the target domain to use as ground truth in subsequent training iterations. However, standard self-training often suffers from two major drawbacks: it discards "hard" samples where the model is uncertain, and it can be misled by "easy" samples that are confidently but incorrectly labeled. This paper introduces a two-stage framework to address these limitations by mining hard samples and implementing an uncertain learning strategy.

Methodology

The proposed framework consists of two distinct rounds of self-training designed to refine pseudo-labels and optimize the network's learning process.

Overview of the two-round self-training framework.

Round 1: Entropy-Ranking for Hard Sample Mining

Instead of relying on a simple confidence threshold that ignores low-confidence pixels, the authors employ Entropy-Ranking. Entropy serves as a proxy for prediction uncertainty. By sorting pixels based on their entropy values, the method distinguishes between "easy" (low entropy) and "hard" (high entropy) samples.

Visualization of entropy-ranking components.

A hyper-parameter $\lambda_r$ is used to define the boundary between these categories. This approach allows the model to systematically refine and "rectify" hard samples, transforming previously discarded data into informative training signals for the second round.

Round 2: Uncertain Learning and Loss Optimization

In the second round, the model reconciles original pseudo-labels with the updated versions from Round 1. To mitigate the impact of potentially false labels, a Specific Uncertain Loss ($L_{seg2}$) is introduced. This loss function incorporates an Uncertain Term ($UC$), which combines entropy and a distance metric between label sets.

If the labels disagree, the $UC$ term increases, and an exponential weight ($e^{-UC}$) reduces the pixel's contribution to the total loss. This mechanism allows the network to dynamically "doubt" its own predictions, preventing incorrect pseudo-labels from corrupting the optimization process.

Experimental Results

The method was rigorously tested on three major UDA benchmarks: GTA5 to Cityscapes, SYNTHIA to Cityscapes, and Cityscapes to Oxford RobotCar.

  • GTA5 → Cityscapes: The baseline mIoU of 43.8% improved to 46.3% after the first round and reached 47.4% after the second round.
  • SYNTHIA → Cityscapes: The method achieved a mIoU of 48.9%, a significant leap from the 41.2% baseline.
  • Qualitative Performance: Visual inspections revealed superior segmentation of complex, thin structures like poles and traffic signs, which are often misclassified by standard UDA methods.

Conclusion

By treating hard samples as valuable assets rather than noise, the proposed entropy-ranking and uncertain learning strategies significantly narrow the intra-domain gap in semantic segmentation. The framework provides a robust way to handle the inherent uncertainty of pseudo-labels, leading to state-of-the-art performance across multiple benchmarks. Future research may focus on automating hyper-parameter selection to further enhance the system's adaptability across diverse environments.

Summary updated

Abstract

Dear Editor, This letter develops two new self-training strategies for domain adaptive semantic segmentation, which formulate self-training into the processes of mining more training samples and reducing influence of the false pseudo-labels. Particularly, a self-training strategy based on entropy-ranking is proposed to mine intra-domain information. Thus, numerous false pseudo-labels can be exploited and rectified, and more pseudo-labels can be involved in training. Meanwhile, another novel self-training strategy is developed to handle the regions that may possess false pseudo-labels. In detail, a specific uncertain loss, that makes the network automatically decide whether the pseudo-labels are true, is proposed to improve the network optimization. Consequently, the influence of false pseudo-labels can be reduced. Experimental results prove that, compared with the baseline, the average mIoU performance gain brought by our method can attain 4.3%. Extensive benchmark experiments further highlight the effectiveness of our method against existing state-of-the-arts.
BibTeX
@article{peng2022domain,
  author = {Chengli Peng and Jiayi Ma},
  title = {{Domain Adaptive Semantic Segmentation via Entropy-Ranking and Uncertain Learning-Based Self-Training}},
  journal = {IEEE CAA J. Autom. Sinica},
  year = {2022},
  url = {https://openreview.net/forum?id=HgMLUveJQ3},
}
Figure