KLIEP-based Density Ratio Estimation for Semantically Consistent Synthetic to Real Images Adaptation in Urban Traffic Scenes

Artem Savkin, Federico TombariIROS 2020 · DOI · OpenReview

Abstract

Synthetic data has been applied in many deep learning based computer vision tasks. Limited performance of algorithms trained solely on synthetic data has been approached with domain adaptation techniques such as the ones based on generative adversarial framework. We demonstrate how adversarial training alone can introduce semantic inconsistencies in translated images. To tackle this issue we propose density prematching strategy using KLIEP-based density ratio estimation procedure. Finally, we show that aforementioned strategy improves quality of translated images of underlying method and their usability for the semantic segmentation task in the context of autonomous driving.

Introduction

Deep learning models for autonomous driving require vast amounts of labeled data, which is often prohibitively expensive or dangerous to collect in the real world. Synthetic data from simulators offers a scalable alternative; however, models trained on synthetic data often fail to generalize to real-world environments due to domain shift. Specifically, covariate shift occurs when the input distribution of synthetic images differs significantly from real-world images.

The Challenge of Domain Shift and Semantic Inconsistency

Standard Unsupervised Domain Adaptation (UDA) techniques frequently employ Generative Adversarial Networks (GANs) to translate synthetic images into the target real-world domain. While effective at matching low-level statistics, GANs can be "over-aggressive," introducing semantic inconsistencies to satisfy the discriminator. For instance, a GAN might erroneously place vegetation in the sky or remove critical objects like cars to make the synthetic scene appear more "real" according to its learned distribution. Such hallucinations render the data unsuitable for safety-critical applications like autonomous driving.

A typical synthetic urban scene used as source data from the SYNTHIA dataset.

Methodology: Density Prematching with KLIEP

To address these inconsistencies, the authors propose a density prematching strategy using importance weighting. The core idea is to weight synthetic samples based on their relevance to the target real-world distribution before or during the adversarial training process.

  1. Importance Weighting: Each synthetic sample is assigned a weight representing the ratio between its probability in the real domain versus the synthetic domain.
  2. KLIEP (Kullback-Leibler Importance Estimation Procedure): Directly estimating high-dimensional densities is computationally difficult. The authors utilize KLIEP to estimate the density ratio directly by minimizing the KL divergence between the true target distribution and the estimated weighted distribution.
  3. Weighted Loss Functions: These importance weights are integrated into the adversarial and cycle-consistency losses of the translation network. This forces the generator to prioritize features that naturally overlap with the real-world domain, preserving the underlying semantic structure without requiring additional semantic labels.

Key Results

The proposed method was evaluated using the SYNTHIA (synthetic) and Cityscapes (real) datasets.

Qualitative Performance

As shown in the visual results, the KLIEP-based adaptation produces cleaner and more semantically consistent images compared to unweighted baselines. By focusing the translation on relevant features, the model avoids the "patchy" hallucinations (such as trees in the sky) common in standard GAN-based translation, resulting in more realistic urban scenes.

Qualitative comparison of synthetic source images (left) and translated real-world style images (right) using the proposed KLIEP-based density ratio estimation.

Quantitative Improvements

The method significantly improved Mean Intersection over Union (mIoU) for semantic segmentation on the Cityscapes validation set:

  • mIoU Gains: Using a DRN26 model, the proposed method achieved a 39.7 mIoU, a substantial increase over the standard CycleGAN baseline (32.4).
  • Class-Specific Accuracy: The "Sky" class, which is particularly prone to semantic errors in standard DA, saw an IoU improvement of nearly 7% compared to other adaptation methods.
  • Toy Dataset Validation: In controlled experiments, the importance-weighted loss improved the Wasserstein distance to the target distribution by 20% and energy distance by 15%.

Conclusion

By incorporating KLIEP-based density ratio estimation, the authors demonstrate a robust method for improving the semantic consistency of synthetic-to-real image translation. This importance-weighting strategy effectively mitigates the risks of adversarial training "hallucinations," leading to more reliable training data for autonomous driving systems. Future work may explore automated kernel tuning for KLIEP and the application of this technique to other domains like medical imaging or satellite data.

Summary updated

Abstract

Synthetic data has been applied in many deep learning based computer vision tasks. Limited performance of algorithms trained solely on synthetic data has been approached with domain adaptation techniques such as the ones based on generative adversarial framework. We demonstrate how adversarial training alone can introduce semantic inconsistencies in translated images. To tackle this issue we propose density prematching strategy using KLIEP-based density ratio estimation procedure. Finally, we show that aforementioned strategy improves quality of translated images of underlying method and their usability for the semantic segmentation task in the context of autonomous driving.
BibTeX
@inproceedings{savkin2020kliep,
  author = {Artem Savkin and Federico Tombari},
  title = {{KLIEP-based Density Ratio Estimation for Semantically Consistent Synthetic to Real Images Adaptation in Urban Traffic Scenes}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2020},
  doi = {10.1109/iros45743.2020.9341547},
}
Figure