Abstract
In this paper, we consider the problem of domain generalization in semantic segmentation, which aims to learn a robust model using only labeled synthetic (source) data. The model is expected to perform well on unseen real (target) domains. Our study finds that the image style variation can largely influence the model's performance and the style features can be well represented by the channel-wise mean and standard deviation of images. Inspired by this, we propose a novel adversarial style augmentation (AdvStyle) approach, which can dynamically generate hard stylized images during training and thus can effectively prevent the model from overfitting on the source domain. Specifically, AdvStyle regards the style feature as a learnable parameter and updates it by adversarial training. The learned adversarial style feature is used to construct an adversarial image for robust model training. AdvStyle is easy to implement and can be readily applied to different models. Experiments on two synthetic-to-real semantic segmentation benchmarks demonstrate that AdvStyle can significantly improve the model performance on unseen real domains and show that we can achieve the state of the art. Moreover, AdvStyle can be employed to domain generalized image classification and produces a clear improvement on the considered datasets.
Introduction: The Challenge of Domain Generalization
Semantic segmentation—the task of identifying every pixel in an image—is a cornerstone of autonomous driving. However, deep learning models are notoriously fragile when faced with the "domain gap." A model trained on synthetic data from simulators like GTA V often fails in real-world environments due to differences in lighting, weather, and camera hardware.
Existing solutions often struggle to capture the sheer variety of real-world conditions. This paper, presented at NeurIPS 2022, introduces Adversarial Style Augmentation (AdvStyle), a novel training strategy that makes models more robust by training them against an "adversarial stylist."
Adversarial Style Augmentation (AdvStyle)
The core insight of AdvStyle is that an image's style can be mathematically represented by its channel-wise mean and standard deviation. Instead of using fixed augmentations, AdvStyle treats these style features as learnable parameters during training.
AdvStyle operates in a two-step loop:
- Adversarial Style Learning: The system identifies the "hardest" possible style for a given image—the one that maximizes the model's current segmentation error. It asks: "What is the most confusing way I could recolor this image to make the model fail?"
- Robust Model Training: The model is then trained on this adversarial version. Because AdvStyle modifies global statistics rather than individual pixels, it preserves the semantic consistency (shapes and boundaries) of the scene, which is crucial for pixel-perfect segmentation.
Experimental Setup and Datasets
The researchers focused on the synthetic-to-real generalization task, training on synthetic datasets and evaluating on diverse real-world scenes:
- Source (Training): GTAV (24,966 images) and SYNTHIA (9,400 images).
- Target (Evaluation): Cityscapes (Germany), BDD-100K (US), and Mapillary (Global).
- Classification: The method was also validated on the Digits and PACS datasets to prove its versatility beyond segmentation.
Key Results and Insights
AdvStyle consistently outperformed previous state-of-the-art methods across multiple benchmarks:
- Significant mIoU Gains: On a ResNet-50 backbone trained on GTAV, AdvStyle improved the mean Intersection over Union (mIoU) across three real-world datasets from 27.42% to 37.39%.
- Synergy with Existing Methods: When combined with other techniques like IBN-Net or Instance Selective Whitening (ISW), AdvStyle provided further boosts, reaching over 42% mIoU.
- Generalization to Classification: On the PACS dataset, adding AdvStyle improved average accuracy from 58.9% to 65.4%.
Conclusion
AdvStyle offers a powerful, model-agnostic way to improve domain generalization. While it roughly doubles training time due to the extra adversarial step, it significantly bridges the gap between simulation and reality. Future work may address structural domain shifts—such as differences in architecture or vehicle types—that style augmentation alone cannot resolve.
Summary updated