Abstract
Unsupervised domain adaptation (UDA) for semantic segmentation has been attracting attention recently, as it could be beneficial for various label-scarce real-world scenarios. Despite significant progress, current works mainly focus on a single-source single-target setting, which cannot handle more practical settings of multiple targets or even unseen targets. In this paper, we investigate open compound domain adaptation (OCDA), which deals with mixed and novel situations at the same time. We present a novel framework based on three main design principles: discover, hallucinate, and adapt. The scheme first clusters compound target data based on style, discovering multiple latent domains (discover). Then, it hallucinates multiple latent target domains in the source by using image-translation (hallucinate). Finally, target-to-source alignment is learned separately between domains (adapt). Our solution replaces a hard OCDA problem with much easier multiple UDA problems. We evaluate our solution on the standard benchmark GTA to C-driving and achieved new state-of-the-art results.
Introduction
Most AI models for semantic segmentation are trained on synthetic data because labeling real-world pixels is incredibly expensive. While traditional Unsupervised Domain Adaptation (UDA) excels at moving from a single synthetic source to a single real-world target, it often fails when the target is a mixture of conditions or contains surprises the model never saw during training.
Standard UDA assumes the target domain is homogeneous. In reality, a "target" like "driving in a city" includes rain, snow, clouds, and night. This is a Compound Domain. Furthermore, at test time, the model might encounter an Open Domain—a condition it never saw during the adaptation phase (like an overcast day). Existing methods often suffer from biased alignment: the model focuses on the "easy" target samples that already look like the source, failing on "hard" samples like night-time or heavy rain.
Methodology: The DHA Framework
The researchers from KAIST propose a framework called DHA (Discover, Hallucinate, and Adapt) to bridge this gap by breaking one hard problem into several easier ones.
- Discover: Since labels for specific conditions (e.g., "rainy" or "snowy") are unavailable, the model uses style-based clustering. By analyzing convolutional feature statistics (mean and variance), it automatically partitions the unlabeled target data into $K$ latent domains.
- Hallucinate: To facilitate learning, the researchers use an image-to-image translation network to "hallucinate" these discovered styles onto the source images. If the model discovered a "rainy" latent domain, it transforms synthetic GTA5 images to look rainy. A style consistency loss ensures the translated images match the target style without collapsing into a generic look.
- Adapt: Instead of using one big discriminator to align all data at once, DHA uses $K$ separate discriminators—one for each latent domain. This "divide and conquer" approach ensures that the model learns to align specific source-target pairs (e.g., "hallucinated rain" to "real rain") without the easy domains drowning out the hard ones.
Key Results
The primary benchmark is GTA5 to C-driving, where the source is synthetic GTA5 images and the compound target includes Rainy, Snowy, and Cloudy sequences from the BDD100K dataset. The "Open" target consists of unseen Overcast sequences.
- State-of-the-Art Performance: On the GTA5 to C-driving benchmark, DHA achieved a mean Intersection-over-Union (mIoU) of 32.3, significantly outperforming the previous best OCDA method (25.0) and standard UDA methods like AdaptSeg (22.5).
- Robustness to the Unknown: Even on the "Open" (overcast) domain, DHA showed superior generalization, proving that latent style discovery helps the model handle unseen but related conditions.
- Qualitative Improvement: Visual results show much cleaner segmentation in "hard" conditions where previous models produced "noisy" or completely incorrect classifications, such as mistaking a dark road for a sidewalk in rainy or night-time conditions.
Conclusion
The DHA framework demonstrates that decomposing complex domain adaptation into style-specific sub-problems is a powerful strategy for real-world computer vision. By explicitly creating "paired" versions of the source data for every target style, the problem reverts to a set of manageable UDA tasks. While the method requires tuning the number of clusters ($K$) and introduces some computational overhead due to multiple discriminators, it effectively prevents biased alignment and sets a new standard for robust semantic segmentation in diverse environments.
Summary updated