Uncertainty Reduction for Model Adaptation in Semantic Segmentation

Prabhu Teja Sivaprasad, François FleuretCVPR 2021 · OpenReview

Abstract

Traditional methods for Unsupervised Domain Adaptation (UDA) targeting semantic segmentation exploit information common to the source and target domains, using both labeled source data and unlabeled target data. In this paper, we investigate a setting where the source data is unavailable, but the classifier trained on the source data is; hence named "model adaptation". Such a scenario arises when data sharing is prohibited, for instance, because of privacy, or Intellectual Property (IP) issues. To tackle this problem, we propose a method that reduces the uncertainty of predictions on the target domain data. We accomplish this in two ways: minimizing the entropy of the predicted posterior, and maximizing the noise robustness of the feature representation. We show the efficacy of our method on the transfer of segmentation from computer generated images to real-world driving images, and transfer between data collected in different cities, and surprisingly reach performance competitive with that of the methods that have access to source data.

Introduction

In Unsupervised Domain Adaptation (UDA), the goal is to take a model trained on a "source" domain (e.g., synthetic video game images) and make it work on a "target" domain (e.g., real-world driving footage). Traditionally, this requires having both datasets side-by-side during the adaptation process. However, in many industries—like healthcare or autonomous driving—sharing the source data is impossible due to privacy laws (GDPR/HIPAA), intellectual property concerns, or sheer data volume.

This paper addresses the "model adaptation" setting: you have a pre-trained classifier and a pile of unlabeled target data, but the source pixels are gone. The challenge is to prevent the model from making "confident but wrong" predictions caused by the domain shift.

Proposed Method

The authors' core philosophy is that a well-adapted model should produce stable, certain predictions on the target data. They implement this through three main pillars:

  1. Entropy Regularization ($L_{ent}$): A standard technique that penalizes the model for being "unsure" (having high entropy in its probability distribution). This forces the model to commit to specific class predictions.
  2. Noise Robustness via Auxiliary Decoders ($L_{un}$): This is the paper's key innovation. They introduce "LastLayer-Dropout" between the feature extractor (backbone) and the decoder. They attach multiple auxiliary decoders that see different "corrupted" versions of the features. By forcing these decoders to agree with the main decoder's prediction, the network learns to produce feature representations that are stable and robust to noise.
  3. Class-Balanced Pseudo-Labeling ($L_{PL}$): The model treats its own high-confidence predictions as "ground truth" labels to self-train. To prevent the model from only learning common classes (like "Road" or "Sky"), they use a threshold that varies per class, ensuring that rarer objects (like "Bicycles") are still included in the training signal.
The proposed architecture featuring a shared backbone and multiple auxiliary decoders with dropout to enforce noise robustness.

Experimental Results

The researchers evaluated their method on three standard benchmarks for domain adaptation in semantic segmentation: GTA5 $\rightarrow$ Cityscapes, Synthia $\rightarrow$ Cityscapes, and Cityscapes $\rightarrow$ NTHU Crosscity.

The proposed method reached performance levels that were "at-par or better" than classic UDA methods that did have access to source data. For example, in the GTA5 $\rightarrow$ Cityscapes task, the model achieved a 45.1 mIoU (mean Intersection over Union), outperforming several source-dependent methods like AdaptSegNet (42.4). In cross-city adaptation (e.g., Rome), the model improved from a baseline of 51.0 mIoU to 53.8 mIoU without seeing a single label from the target city.

Visualization of feature distributions across different adaptation stages. The proposed method (e) shows better class separation on target features compared to simple entropy minimization (d) or the source-trained baseline (c).

Key Insights and Conclusion

A primary takeaway is that uncertainty serves as an excellent proxy for labels. When ground truth is absent, the "sharpness" and "stability" of a model's predictions indicate how well it understands the new domain. However, stability is more important than mere confidence; simply minimizing entropy can lead to a model that is confidently wrong. The addition of the noise-robustness loss ($L_{un}$) ensures the model finds a robust geometric separation in the feature space.

While the method excels at large background classes, it still struggles with very small, infrequent objects where the initial source model's predictions are too unreliable to bootstrap from. Nevertheless, the results demonstrate that the "information payload" within a pre-trained model is often sufficient for adaptation, making source-free model adaptation a viable alternative to traditional UDA.

Summary updated

Abstract

Traditional methods for Unsupervised Domain Adaptation (UDA) targeting semantic segmentation exploit information common to the source and target domains, using both labeled source data and unlabeled target data. In this paper, we investigate a setting where the source data is unavailable, but the classifier trained on the source data is; hence named ""model adaptation"". Such a scenario arises when data sharing is prohibited, for instance, because of privacy, or Intellectual Property (IP) issues. To tackle this problem, we propose a method that reduces the uncertainty of predictions on the target domain data. We accomplish this in two ways: minimizing the entropy of the predicted posterior, and maximizing the noise robustness of the feature representation. We show the efficacy of our method on the transfer of segmentation from computer generated images to real-world driving images, and transfer between data collected in different cities, and surprisingly reach performance competitive with that of the methods that have access to source data.
BibTeX
@inproceedings{sivaprasad2021uncertainty,
  author = {Prabhu Teja Sivaprasad and François Fleuret},
  title = {{Uncertainty Reduction for Model Adaptation in Semantic Segmentation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  url = {https://openreview.net/forum?id=JzKrDsWXVu},
}
Figure