SPIGAN: Privileged Adversarial Learning from Simulation

Kuan-Hui Lee, Germán Ros, Jie Li, Adrien GaidonICLR 2019

Introduction

Deep learning models for computer vision thrive on massive amounts of data. However, acquiring pixel-perfect annotations for tasks like semantic segmentation—where every pixel must be classified (e.g., as a car, road, or sidewalk)—is prohibitively expensive and time-consuming in the real world. Photo-realistic simulators offer a compelling alternative by generating infinite synthetic data with perfect, free labels.

Despite this advantage, a significant hurdle remains: the domain gap. Synthetic images differ from real-world images in lighting, texture, and overall style. Consequently, models trained exclusively on synthetic data often suffer severe performance drops when deployed in real-world environments. Current Unsupervised Domain Adaptation (UDA) methods attempt to bridge this gap using Generative Adversarial Networks (GANs) to translate synthetic images into realistic ones. Unfortunately, these GANs frequently "hallucinate" or destroy crucial structural details because they lack a true understanding of the scene's underlying 3D geometry.

Visualizing the domain gap and adaptation between synthetic and real-world urban scenes.

The SPIGAN Approach

To address the structural degradation caused by standard GANs, the authors introduce SPIGAN (Simulator Privileged Information GAN). The core insight of SPIGAN is that simulators are not merely "black boxes" that render images; they possess rich internal knowledge of the simulated world's physics and geometry. The authors term this hidden data Privileged Information (PI) and specifically leverage the simulator's z-buffer (depth map) to guide the learning process.

SPIGAN simultaneously trains four distinct networks:

  1. Generator ($G$): Transforms synthetic images to mimic the style of real-world images.
  2. Discriminator ($D$): Attempts to distinguish between the Generator's "adapted" synthetic images and actual real-world images.
  3. Task Network ($T$): The primary objective network, tasked with performing semantic segmentation on real images.
  4. Privileged Network ($P$): A novel addition trained to predict the simulator's original depth map from both the raw synthetic image and the Generator's "real-looking" adapted image.
Depth maps serve as Privileged Information (PI) to enforce structural consistency during image translation.

Why this works: The Privileged Network acts as a strict structural anchor. If the Generator attempts to make an image look "real" by hallucinating a building out of existence or distorting the shape of a road, the Privileged Network will fail to predict the correct underlying depth. This forces the Generator to alter only the style (such as textures and lighting) while rigorously preserving the content (geometry and object placement). Interestingly, the authors found that these structural constraints from the PI network also made the notoriously unstable GAN training process significantly more stable.

Experimental Setup and Results

The researchers evaluated SPIGAN across three major datasets:

  • SYNTHIA (Source): A large-scale synthetic dataset of urban scenes providing full labels and depth information (PI) for training.
  • Cityscapes (Target): A standard real-world dataset of German cities, used as unlabeled target data during training.
  • Mapillary Vistas (Target): A highly diverse and challenging real-world dataset featuring global images under various weather conditions.

Key Performance Gains

SPIGAN successfully outperformed existing state-of-the-art unsupervised domain adaptation methods:

  • Cityscapes: The model achieved a 3% improvement in mean Intersection-over-Union (mIoU) at lower resolutions and a 1% improvement at higher resolutions compared to baselines lacking Privileged Information.
  • Mapillary Vistas: On this significantly harder dataset, SPIGAN demonstrated remarkable robustness. While other adaptation methods often suffered from "negative transfer"—where the adaptation process actually degrades model performance—SPIGAN's depth constraints kept the learning trajectory on track.
  • Qualitative Improvements: The model showed marked improvements in identifying structural "layout" classes (like sidewalks and roads) and distinct objects (such as riders and bicycles), which standard adaptation methods frequently blur together.

Conclusion and Future Directions

SPIGAN demonstrates that we can move beyond treating simulators as simple image generators by exploiting their internal state to guide domain adaptation. By using geometry as a regularizer, SPIGAN preserves semantic consistency and prevents the visual artifacts common in standard GAN translations.

While SPIGAN successfully narrows the sim-to-real gap, a performance disparity still exists compared to models trained on manually labeled real-world data. Furthermore, this study primarily focused on depth (z-buffer) as the sole source of Privileged Information. Future work could explore other rich simulator outputs—such as optical flow or surface normals—and apply the SPIGAN framework to broader autonomous vehicle tasks like object detection or path planning.

Summary updated

BibTeX
@inproceedings{lee2019spigan,
  author = {Kuan-Hui Lee and Germán Ros and Jie Li and Adrien Gaidon},
  title = {{SPIGAN: Privileged Adversarial Learning from Simulation}},
  booktitle = {International Conference on Learning Representations},
  year = {2019},
}
Figure