Unsupervised Domain Adaptation in Semantic Segmentation: a Review

Marco Toldo, Andrea Maracani, Umberto Michieli, Pietro Zanuttigh

Abstract

The aim of this paper is to give an overview of the recent advancements in the Unsupervised Domain Adaptation (UDA) of deep networks for semantic segmentation. This task is attracting a wide interest, since semantic segmentation models require a huge amount of labeled data and the lack of data fitting specific requirements is the main limitation in the deployment of these techniques. This problem has been recently explored and has rapidly grown with a large number of ad-hoc approaches. This motivates us to build a comprehensive overview of the proposed methodologies and to provide a clear categorization. In this paper, we start by introducing the problem, its formulation and the various scenarios that can be considered. Then, we introduce the different levels at which adaptation strategies may be applied: namely, at the input (image) level, at the internal features representation and at the output level. Furthermore, we present a detailed overview of the literature in the field, dividing previous methods based on the following (non mutually exclusive) categories: adversarial learning, generative-based, analysis of the classifier discrepancies, self-teaching, entropy minimization, curriculum learning and multi-task learning. Novel research directions are also briefly introduced to give a hint of interesting open problems in the field. Finally, a comparison of the performance of the various methods in the widely used autonomous driving scenario is presented.

The Challenge of Data-Hungry Segmentation

Deep learning has revolutionized computer vision, but it suffers from a significant "brittleness" problem: models trained on one dataset (the source domain) often fail when deployed in slightly different environments (the target domain). This phenomenon is known as domain shift.

In semantic segmentation—where every pixel must be classified into a category—the problem is particularly acute. Creating ground-truth labels for this task is incredibly slow and expensive, often requiring up to 90 minutes per image. While synthetic engines like GTA5 or CARLA can generate infinite labeled data, a model trained on "video game pixels" does not naturally generalize to real-world sensor data. Unsupervised Domain Adaptation (UDA) seeks to bridge this gap, allowing models to learn from synthetic data while performing accurately on unlabeled real-world data.

Comparison of computer vision tasks, highlighting the complexity of semantic segmentation where every pixel is classified.

Taxonomy of Adaptation Strategies

UDA strategies are generally categorized by where the adaptation occurs within the neural network pipeline:

  • Input-level Adaptation: Employs Generative Adversarial Networks (GANs) or style transfer (e.g., CycleGAN) to translate source images into the visual style of the target domain before they reach the segmentation network.
  • Feature-level Adaptation: Forces the network's internal encoder to extract domain-invariant features. By using a discriminator to ensure feature maps from different domains are indistinguishable, the network learns to ignore domain-specific noise.
  • Output-level Adaptation: Aligns the final prediction maps. Since segmentation outputs share consistent spatial structures across domains (e.g., the sky is typically at the top of an image), this space is often easier to align than high-dimensional internal features.

Beyond these levels, the literature identifies several algorithmic pillars, including Adversarial Learning, Self-teaching (using high-confidence pseudo-labels), and Entropy Minimization, which forces the model to make more certain predictions on the target data.

Application Domains and Benchmarks

While UDA is applicable across various fields, the autonomous driving scenario has become the de-facto standard for benchmarking. This is driven by the availability of high-quality synthetic datasets like GTA5 and SYNTHIA, and real-world targets like Cityscapes and Mapillary Vistas.

UDA is critical for diverse real-world applications including autonomous driving, warehouse logistics, and domestic robotics.

Key Insights

  • Output-level Efficacy: Adaptation at the output level is often more effective than at the feature level because the output space is lower-dimensional and retains scene structural layouts that are consistent across domains.
  • The Backbone Matters: Models utilizing ResNet-101 backbones significantly outperform those using older architectures like VGG-16, often by margins of 10-15% mIoU.
  • The Performance Gap: Despite rapid progress, a gap remains. Top UDA methods on the GTA5 to Cityscapes task achieve approximately 50% mIoU, whereas fully supervised models trained directly on target data reach 70-80%.

Future Directions and Open Challenges

Future research is shifting toward more complex scenarios. Continual Learning is vital for models that must adapt to changing conditions (e.g., weather or time of day) without forgetting previous knowledge. Furthermore, moving beyond "Closed Set" assumptions to Open Set or Boundless DA—where the target domain may contain entirely new object classes—remains a major frontier in the field. Addressing the high computational cost of generative methods also remains a priority for practical deployment.

Summary updated

Abstract

The aim of this paper is to give an overview of the recent advancements in the Unsupervised Domain Adaptation (UDA) of deep networks for semantic segmentation. This task is attracting a wide interest, since semantic segmentation models require a huge amount of labeled data and the lack of data fitting specific requirements is the main limitation in the deployment of these techniques. This problem has been recently explored and has rapidly grown with a large number of ad-hoc approaches. This motivates us to build a comprehensive overview of the proposed methodologies and to provide a clear categorization. In this paper, we start by introducing the problem, its formulation and the various scenarios that can be considered. Then, we introduce the different levels at which adaptation strategies may be applied: namely, at the input (image) level, at the internal features representation and at the output level. Furthermore, we present a detailed overview of the literature in the field, dividing previous methods based on the following (non mutually exclusive) categories: adversarial learning, generative-based, analysis of the classifier discrepancies, self-teaching, entropy minimization, curriculum learning and multi-task learning. Novel research directions are also briefly introduced to give a hint of interesting open problems in the field. Finally, a comparison of the performance of the various methods in the widely used autonomous driving scenario is presented.
BibTeX
@inproceedings{toldo2020unsupervised,
  author = {Marco Toldo and Andrea Maracani and Umberto Michieli and Pietro Zanuttigh},
  title = {{Unsupervised Domain Adaptation in Semantic Segmentation: a Review}},
  booktitle = {Technologies},
  year = {2020},
  doi = {10.3390/technologies8020035},
}
Figure