Discover, Hallucinate, and Adapt: Open Compound Domain Adaptation for Semantic Segmentation

KwanYong Park, Sanghyun Woo, Inkyu Shin, In So Kweon

Abstract

Unsupervised domain adaptation (UDA) for semantic segmentation has been attracting attention recently, as it could be beneficial for various label-scarce real-world scenarios. Despite significant progress, current works mainly focus on a single-source single-target setting, which cannot handle more practical settings of multiple targets or even unseen targets. In this paper, we investigate open compound domain adaptation (OCDA), which deals with mixed and novel situations at the same time. We present a novel framework based on three main design principles: discover, hallucinate, and adapt. The scheme first clusters compound target data based on style, discovering multiple latent domains (discover). Then, it hallucinates multiple latent target domains in the source by using image-translation (hallucinate). Finally, target-to-source alignment is learned separately between domains (adapt). Our solution replaces a hard OCDA problem with much easier multiple UDA problems. We evaluate our solution on the standard benchmark GTA to C-driving and achieved new state-of-the-art results.

Introduction

Most AI models for semantic segmentation are trained on synthetic data because labeling real-world pixels is incredibly expensive. While traditional Unsupervised Domain Adaptation (UDA) excels at moving from a single synthetic source to a single real-world target, it often fails when the target is a mixture of conditions or contains surprises the model never saw during training.

Standard UDA assumes the target domain is homogeneous. In reality, a "target" like "driving in a city" includes rain, snow, clouds, and night. This is a Compound Domain. Furthermore, at test time, the model might encounter an Open Domain—a condition it never saw during the adaptation phase (like an overcast day). Existing methods often suffer from biased alignment: the model focuses on the "easy" target samples that already look like the source, failing on "hard" samples like night-time or heavy rain.

Methodology: The DHA Framework

The researchers from KAIST propose a framework called DHA (Discover, Hallucinate, and Adapt) to bridge this gap by breaking one hard problem into several easier ones.

Conceptual overview of the DHA framework compared to traditional UDA.
  1. Discover: Since labels for specific conditions (e.g., "rainy" or "snowy") are unavailable, the model uses style-based clustering. By analyzing convolutional feature statistics (mean and variance), it automatically partitions the unlabeled target data into $K$ latent domains.
  2. Hallucinate: To facilitate learning, the researchers use an image-to-image translation network to "hallucinate" these discovered styles onto the source images. If the model discovered a "rainy" latent domain, it transforms synthetic GTA5 images to look rainy. A style consistency loss ensures the translated images match the target style without collapsing into a generic look.
  3. Adapt: Instead of using one big discriminator to align all data at once, DHA uses $K$ separate discriminators—one for each latent domain. This "divide and conquer" approach ensures that the model learns to align specific source-target pairs (e.g., "hallucinated rain" to "real rain") without the easy domains drowning out the hard ones.

Key Results

The primary benchmark is GTA5 to C-driving, where the source is synthetic GTA5 images and the compound target includes Rainy, Snowy, and Cloudy sequences from the BDD100K dataset. The "Open" target consists of unseen Overcast sequences.

Qualitative comparison of semantic segmentation results across different weather conditions.
  • State-of-the-Art Performance: On the GTA5 to C-driving benchmark, DHA achieved a mean Intersection-over-Union (mIoU) of 32.3, significantly outperforming the previous best OCDA method (25.0) and standard UDA methods like AdaptSeg (22.5).
  • Robustness to the Unknown: Even on the "Open" (overcast) domain, DHA showed superior generalization, proving that latent style discovery helps the model handle unseen but related conditions.
  • Qualitative Improvement: Visual results show much cleaner segmentation in "hard" conditions where previous models produced "noisy" or completely incorrect classifications, such as mistaking a dark road for a sidewalk in rainy or night-time conditions.

Conclusion

The DHA framework demonstrates that decomposing complex domain adaptation into style-specific sub-problems is a powerful strategy for real-world computer vision. By explicitly creating "paired" versions of the source data for every target style, the problem reverts to a set of manageable UDA tasks. While the method requires tuning the number of clusters ($K$) and introduces some computational overhead due to multiple discriminators, it effectively prevents biased alignment and sets a new standard for robust semantic segmentation in diverse environments.

Summary updated

Abstract

Unsupervised domain adaptation (UDA) for semantic segmentation has been attracting attention recently, as it could be beneficial for various label-scarce real-world scenarios (e.g., robot control, autonomous driving, medical imaging, etc.). Despite the significant progress in this field, current works mainly focus on a single-source single-target setting, which cannot handle more practical settings of multiple targets or even unseen targets. In this paper, we investigate open compound domain adaptation (OCDA), which deals with mixed and novel situations at the same time, for semantic segmentation. We present a novel framework based on three main design principles: discover, hallucinate, and adapt. The scheme first clusters compound target data based on style, discovering multiple latent domains (discover). Then, it hallucinates multiple latent target domains in source by using image-translation (hallucinate). This step ensures the latent domains in the source and the target to be paired. Finally, target-to-source alignment is learned separately between domains (adapt). In high-level, our solution replaces a hard OCDA problem with much easier multiple UDA problems. We evaluate our solution on standard benchmark GTA to C-driving, and achieved new state-of-the-art results.
BibTeX
@inproceedings{park2020discover,
  author = {KwanYong Park and Sanghyun Woo and Inkyu Shin and In So Kweon},
  title = {{Discover, Hallucinate, and Adapt: Open Compound Domain Adaptation for Semantic Segmentation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=9FYNPzUrBqM},
}
Figure