Empirical Generalization Study: Unsupervised Domain Adaptation vs. Domain Generalization Methods for Semantic Segmentation in the Wild

Fabrizio J. Piva, Daan de Geus, Gijs DubbelmanWACV 2023

TL;DR

This paper tackles the critical challenge of making semantic segmentation models robust in real-world driving scenarios, where data distributions can vary wildly. By establishing a fair evaluation framework, they empirically demonstrate that Unsupervised Domain Adaptation (UDA) methods, which leverage unlabeled data, significantly outperform Domain Generalization (DG) methods, achieving performance on par with fully supervised approaches.

What Problem Does This Paper Address?

Autonomous vehicles and robots rely heavily on semantic segmentation to understand their surroundings, classifying every pixel in an image. However, models trained on specific datasets often struggle when deployed in real-world "wild" conditions due to variations in lighting, weather, camera properties, and locations. This performance degradation is a major safety concern. While two main strategies exist to combat this – Domain Generalization (DG) and Unsupervised Domain Adaptation (UDA) – existing research has made direct comparisons difficult due to differing architectures, training settings, and evaluation metrics. This paper aims to bridge this gap by providing a standardized, empirical comparison of UDA and DG methods to determine which approach offers superior generalization capabilities for real-world semantic segmentation.

Methods

The authors propose a novel evaluation framework designed for a fair comparison of UDA and DG methods. Key aspects of their methodology include:

  • Normalized Architecture: All methods are trained and evaluated using a consistent DeepLabv2 architecture with a ResNet-101 backbone, pretrained on ImageNet. This eliminates architectural differences as a confounding factor.
  • Real-World Data Focus: The experiments exclusively use real-world datasets for both training and evaluation, moving away from synthetic-to-real adaptations common in UDA research.
  • Standardized Training and Evaluation: A consistent set of training and validation splits are used across all experiments. Performance is measured using average mean Intersection over Union (mIoU) on both seen and unseen domains.
  • UDA vs. DG Comparison: The core of the study involves training both UDA and DG methods on the same labeled dataset (Di) and comparing their performance on unseen domains (Du). UDA methods are also allowed to leverage unlabeled data (Dni), while DG methods are not.
  • Impact of Labeled Data Heterogeneity: The study investigates how the homogeneity or heterogeneity of the labeled training data affects the performance of DG and UDA methods.
  • Impact of Unlabeled Data Quantity: The authors analyze the effect of varying the amount of unlabeled data used by UDA methods on generalization performance.

Datasets

The study utilizes several publicly available datasets:

  • Cityscapes (CS): Used as a homogeneous labeled domain (Di) in one experiment.
  • BDD-100K (BDD): Used as an unlabeled domain (Dni).
  • Mapillary Vistas (MAP): Used as a heterogeneous labeled domain (Di) in one experiment and as an unlabeled domain (Dni) in another.
  • WildDash (WILD): Used as an unseen domain (Du).
  • IDD: Used as an unseen domain (Du).
  • KITTI: Used as an unseen domain (Du).

The authors performed histogram analysis on Cityscapes and Mapillary Vistas to support their selection of homogeneous and heterogeneous datasets, respectively.

Key Insights and Ideas

  • UDA Dominance: Unsupervised Domain Adaptation (UDA) methods, particularly ProDA, significantly outperform Domain Generalization (DG) methods in generalizing to unseen real-world datasets.
  • Unlabeled Data is Key: The ability of UDA methods to leverage unlabeled data is crucial for achieving strong generalization, enabling them to match or even surpass fully supervised methods.
  • Real-World Performance: UDA methods can achieve performance on unseen real-world data that is comparable to fully supervised models trained on all available labeled data.
  • DG Struggles with Heterogeneity: DG methods show a notable performance drop when the labeled training domain is heterogeneous, likely because their style-estimation techniques are less effective in such varied conditions.
  • Labeled Data Quality Matters Less Than Quantity/Variety: The specific choice of labeled dataset (homogeneous vs. heterogeneous) has less impact on UDA performance than the availability of diverse data, whether labeled or unlabeled.
  • Efficiency of Unlabeled Data: A significant portion of the generalization gain for UDA can be achieved with a relatively small percentage of unlabeled data, suggesting that collecting unlabeled data is a highly efficient strategy for improving robustness.

Key Results

  • Unseen mIoU Improvement: ProDA (a UDA method) achieved an average mIoU improvement of +14.1 and +4.7 points on unseen domains compared to the baseline, in the homogeneous and heterogeneous labeled domain settings, respectively.
  • On Par with Fully Supervised: ProDA's performance on unseen domains was comparable to, and in some cases better than, a fully supervised model trained on all available labeled data.
  • DG Performance Drop: In the heterogeneous labeled domain setting, DG methods experienced a significant drop in generalization performance compared to the baseline.
  • Unlabeled Data Efficiency: For generalization to unseen domains, ProDA achieved nearly its full performance with just 25% of the unlabeled data, and using 100% of unlabeled data was sometimes better than using 100% of labeled data.

Limitations and Future Work

  • Reproducibility: The authors acknowledge that direct comparisons in prior work were difficult due to varying implementations and settings. Their work aims to rectify this by providing a standardized framework.
  • Future Work: The paper suggests that further research could explore the impact of different unlabeled data sources and quantities, as well as investigate more advanced UDA techniques for even greater generalization. The authors also highlight the potential for avoiding significant labeling effort by leveraging UDA strategies.

Summary updated

BibTeX
@inproceedings{piva2023empirical,
  author = {Fabrizio J. Piva and Daan de Geus and Gijs Dubbelman},
  title = {{Empirical Generalization Study: Unsupervised Domain Adaptation vs. Domain Generalization Methods for Semantic Segmentation in the Wild}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure