Adversarial Style Augmentation for Domain Generalized Urban-Scene Segmentation

Zhun Zhong, Yuyang Zhao, Gim Hee Lee, Nicu Sebe

TL;DR

The authors propose AdvStyle, a novel data augmentation technique that improves domain generalization by treating image style as a learnable, adversarial parameter. By dynamically generating "hard" stylized versions of synthetic training images, the method prevents models from overfitting to synthetic textures and significantly boosts performance on unseen real-world urban environments.

What Problem Does This Paper Address?

Deep learning models for semantic segmentation (labeling every pixel in an image, e.g., "road," "car," "tree") perform exceptionally well when the training and testing data come from the same distribution. However, they often fail when deployed in the real world due to domain shift—differences in weather, lighting, or camera sensors.

t-SNE visualization of feature distributions across different datasets, illustrating the domain shift between synthetic (GTAV) and real-world data.

To avoid expensive manual labeling, researchers often train models on synthetic data (like the GTAV dataset). The gap between "perfect" synthetic pixels and "messy" real-world pixels is huge. Existing solutions try to learn "domain-invariant" features or use extra real-world data, but these can be complex to implement or require access to target-domain images during training, which isn't always possible.

Methods

The core idea of AdvStyle is that an image's "style" can be effectively captured by its channel-wise mean and standard deviation. The authors propose a two-step adversarial training loop:

  1. Adversarial Style Learning: The training image is decomposed into a "normalized image" (content) and a "style feature" (mean and variance). Instead of using fixed styles, AdvStyle treats the style feature as a learnable parameter. It performs a "gradient ascent" step to update this style so that it increases the model's segmentation loss. Essentially, it asks: "What style would make this image hardest for the current model to segment?"
  2. Robust Model Training: The model is then trained using both the original image and this newly generated "hard" adversarial image.

Unlike standard adversarial attacks that perturb every pixel (which can destroy semantic labels), AdvStyle only modifies the global style statistics. This ensures the semantic content (where the road is, where the car is) remains perfectly intact while the "look" of the scene changes drastically.

Datasets

The paper evaluates the method primarily on synthetic-to-real urban scene benchmarks:

  • Source (Synthetic): GTAV and SYNTHIA.
  • Target (Real-world): Cityscapes, BDD-100K, and Mapillary Vistas.
  • Classification (Generalization check): The authors also tested AdvStyle on the Digits (MNIST, SVHN, etc.) and PACS datasets to prove its versatility beyond segmentation.

Key Insights and Ideas

  • Style is Statistics: The paper reinforces the idea that channel-wise mean and variance are powerful proxies for "style." By manipulating these, you can simulate different weather or lighting conditions without needing a complex generative model like a GAN.
Impact of different style variations on mIoU performance, demonstrating that style changes significantly affect model accuracy.
  • Adversarial > Random: While random style augmentation (RandStyle) helps, adversarial style augmentation is much more efficient. It forces the model to constantly confront its own weaknesses rather than wasting time on "easy" style variations.
  • Label Preservation: A major "aha" moment is that global style manipulation is "safer" for segmentation than pixel-wise noise. Pixel-wise attacks often change the identity of small objects, making the original ground-truth labels incorrect. AdvStyle keeps the shapes and boundaries identical.
  • Model Agnostic: AdvStyle is a "plug-and-play" module. It doesn't require changing the architecture of the segmentation model (like DeepLabV3+ or ResNet) and can be combined with other domain generalization techniques.

Key Results

  • Significant Gains: On the GTAV-to-Real benchmark using a ResNet-50 backbone, AdvStyle improved the mean Intersection over Union (mIoU) from a baseline of 27.42% to 37.39%.
  • State-of-the-Art: When combined with other techniques like IBN-Net or ISW, AdvStyle achieved new SOTA results, reaching up to 42.22% mIoU on ResNet-101.
  • Cross-Task Success: In image classification (PACS dataset), adding AdvStyle to a standard baseline (ERM) improved average accuracy from 58.9% to 65.4%, outperforming several specialized domain generalization methods.

Limitations and Future Work

  • Training Overhead: While the authors mention that a single-step adversarial update is efficient, it still effectively doubles the forward/backward passes per iteration compared to standard training.
  • Style Complexity: While mean and variance capture a lot, they don't capture complex textures or local style variations (e.g., the specific texture of wet asphalt vs. dry asphalt).
  • Hyperparameter Sensitivity: The "adversarial learning rate" ($\gamma$) requires tuning; if set too high, the generated styles become unrealistic and can hamper training.
  • Future Directions: The authors suggest that exploring more sophisticated style representations beyond simple channel statistics could further close the synthetic-to-real gap.

Summary updated

Abstract

In this paper, we consider the problem of domain generalization in semantic segmentation, which aims to learn a robust model using only labeled synthetic (source) data. The model is expected to perform well on unseen real (target) domains. Our study finds that the image style variation can largely influence the model's performance and the style features can be well represented by the channel-wise mean and standard deviation of images. Inspired by this, we propose a novel adversarial style augmentation (AdvStyle) approach, which can dynamically generate hard stylized images during training and thus can effectively prevent the model from overfitting on the source domain. Specifically, AdvStyle regards the style feature as a learnable parameter and updates it by adversarial training. The learned adversarial style feature is used to construct an adversarial image for robust model training. AdvStyle is easy to implement and can be readily applied to different models. Experiments on two synthetic-to-real semantic segmentation benchmarks demonstrate that AdvStyle can significantly improve the model performance on unseen real domains and show that we can achieve the state of the art. Moreover, AdvStyle can be employed to domain generalized image classification and produces a clear improvement on the considered datasets.
BibTeX
@inproceedings{zhong2022adversarial,
  author = {Zhun Zhong and Yuyang Zhao and Gim Hee Lee and Nicu Sebe},
  title = {{Adversarial Style Augmentation for Domain Generalized Urban-Scene Segmentation}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=L_sHGieq1D},
}
Figure