Style Mixing and Patchwise Prototypical Matching for One-Shot Unsupervised Domain Adaptive Semantic Segmentation

Xinyi Wu, Zhenyao Wu, Yuhang Lu, Lili Ju, Song WangAAAI 2022 · DOI

Introduction

Semantic segmentation—the task of labeling every pixel in an image (e.g., "road," "car," "tree")—is a cornerstone of computer vision for autonomous systems. However, models trained on synthetic data (like GTA5) often struggle when deployed in real-world environments (like Cityscapes) due to "domain shift" caused by differences in lighting, texture, and weather. While Unsupervised Domain Adaptation (UDA) typically requires large unlabeled datasets from the target domain, this paper explores the more constrained "One-Shot" OSUDA setting, where only a single unlabeled target image is available.

The Challenge of One-Shot Adaptation

Traditional UDA methods rely on a diverse set of target images to align feature distributions. In contrast, OSUDA must bridge the domain gap using minimal information. Previous attempts often utilized complex, pre-trained style-transfer modules or memory-intensive adversarial training. The authors propose a more streamlined approach that integrates adaptation directly into the segmentation network, significantly reducing computational overhead.

Comparison of different domain adaptation paradigms: (a) UDA, (b) Domain Generalization, (c) OSUDA via Adversarial Style Mining, and (d) the proposed OSUDA method.

Proposed Methodology

The authors introduce two primary innovations to tackle the one-shot bottleneck without needing extra parameters or heavy external modules: the Style-Mixing Segmentor and Patchwise Prototypical Matching (PPM).

Style-Mixing Segmentor

Instead of using a separate GAN or style-transfer network, the authors embed style-mixing layers directly into the segmentation model. These layers take the feature statistics (mean and variance) of the single target image and "mix" them into the source (synthetic) images during training. By adding a small amount of random noise to these statistics, they can synthesize a variety of "intermediate" domains. This effectively augments the single target image into a broader distribution, helping the model learn features that are invariant to the specific style of the target domain.

Conceptual illustration of domain adaptation: source features (S) are transformed into intermediate domains to bridge the gap toward the target distribution (T).

Patchwise Prototypical Matching (PPM)

Not all parts of a synthetic image are helpful for adapting to a specific real-world image. To avoid "negative adaptation," the authors break the target image's features into small patches (e.g., 32x32) and calculate a "prototype" (an average feature vector) for each. During training, the model compares source pixels to these target prototypes. Pixels that are more similar to the target patches are given higher weight in the loss function, while dissimilar pixels are down-weighted. This provides a "Goldilocks" level of granularity—finer than whole images but less noisy than individual pixels.

Experimental Results

The researchers evaluated their method on three challenging benchmarks:

  • GTA5 → Cityscapes: Achieved a Mean Intersection over Union (mIoU) of 42.8%, rivaling much more complex methods like ASM while being significantly faster.
  • SYNTHIA → Cityscapes: Reached 47.3% mIoU, a significant 6.6% lead over the previous one-shot state-of-the-art.
  • Cityscapes → Dark Zurich: Successfully adapted a daytime model to nighttime using just one image, improving mIoU from 14.6% to 17.5%.

A standout feature of this approach is its efficiency. While previous SOTA methods required hundreds of thousands of iterations and massive GPU memory (~25GB), this method achieves comparable results in just 500 iterations (roughly 20 minutes) on a single consumer GPU using only 10GB of VRAM.

Conclusion

By moving away from external style-transfer modules and focusing on internal feature manipulation and patch-level matching, the authors demonstrate that highly effective one-shot domain adaptation is possible with minimal resources. While the method's success depends on how representative the single target image is, its efficiency and performance make it a robust solution for low-resource environments and rapid deployment.

Summary updated

BibTeX
@inproceedings{wu2022style,
  author = {Xinyi Wu and Zhenyao Wu and Yuhang Lu and Lili Ju and Song Wang},
  title = {{Style Mixing and Patchwise Prototypical Matching for One-Shot Unsupervised Domain Adaptive Semantic Segmentation}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i3.20177},
}
Figure