Synthetic Dataset Acquisition for a Specific Target Domain

Joshua Niemeijer, Sudhanshu Mittal, Thomas BroxWorkshops @ ICCV 2023 · DOI

Abstract

Training deep learning models for semantic segmentation is traditionally labor-intensive, requiring extensive manual annotation of real-world imagery. While synthetic data from simulators offers a scalable alternative with automated labeling, it introduces challenges related to data redundancy and the "reality gap." This paper explores intelligent acquisition strategies to select the most effective synthetic images for training. Our findings demonstrate that selecting a strategic 5% subset of synthetic data can achieve 95.6% of the performance of the full dataset. Furthermore, in adverse real-world conditions, targeted sampling can outperform training on the entire synthetic set by avoiding simulation bias.

Introduction

Semantic segmentation—the task of identifying every pixel in an image—is a cornerstone of autonomous systems. However, annotating a single real-world frame can take up to 90 minutes. Synthetic data from simulators like GTA-5 or CARLA provides a solution, yet it presents two primary obstacles: the "Firehose Problem," where infinite data generation leads to computational redundancy, and the "Reality Gap," where synthetic biases degrade real-world performance. This research investigates whether intelligent sampling can identify the specific synthetic images that best bridge the gap to a target real-world environment.

Methodology

The proposed framework utilizes a simulation engine that populates an image buffer, which is then filtered by an acquisition function to select training samples. Two categories of strategies were evaluated:

  1. Simulation-only Functions: These prioritize internal data characteristics, such as Class Uniform Sampling (balancing rare objects like traffic lights) and Entropy/CoreSet (selecting images based on model uncertainty or set diversity).
  2. Correspondence-based Functions: This novel approach identifies synthetic "twins" of real-world "corner cases"—scenarios where the model currently struggles in the target domain. By using Cosine Similarity in the feature space, the system selects synthetic data that directly addresses real-world failures.
Overview of the proposed pipeline for intelligent synthetic data acquisition, highlighting the use of Unsupervised Domain Adaptation (UDA) and correspondence matching to real-world corner cases.

Experimental Setup

The study utilized three primary datasets for "Synthetic-to-Real" adaptation:

  • GTA-5 (Source): 25,000 synthetic frames serving as the simulation environment.
  • Cityscapes (Target): Real-world European street scenes under favorable weather conditions.
  • ACDC (Target): A challenging dataset featuring adverse conditions such as fog, rain, snow, and nighttime.

Key Results and Discussion

The Efficiency of Intelligent Sampling

On the Cityscapes dataset, the intelligent acquisition strategy achieved a Mean Intersection over Union (mIoU) of 66.17% using only 5% of the available synthetic data (approximately 1,250 frames). This represents 95.6% of the performance achieved by the full dataset, highlighting the significant redundancy in raw simulation output.

The "Less is More" Paradox

In tests involving the ACDC dataset, training on 100% of the synthetic data resulted in lower accuracy than training on a targeted 5% subset. Specifically, the "uni CC max sim" strategy—which combines class balancing with real-world corner-case matching—achieved an mIoU of 49.3%, surpassing the 46.37% achieved by the full dataset. This suggests that large-scale synthetic data can contain "toxic" information or biases that confuse models when applied to complex, adverse real-world scenes.

Class-wise performance metrics across various acquisition strategies, demonstrating the impact of sampling on specific object recognition and overall mIoU.

Conclusion and Future Work

Intelligent synthetic data acquisition effectively mitigates the computational costs of simulation while improving model robustness in challenging environments. However, the success of correspondence-based sampling remains dependent on the quality of the feature encoder. Future research will focus on exploring alternative feature space embeddings and validating these acquisition functions across a broader range of diverse real-world camera systems and environmental conditions.

Summary updated

BibTeX
@inproceedings{niemeijer2023synthetic,
  author = {Joshua Niemeijer and Sudhanshu Mittal and Thomas Brox},
  title = {{Synthetic Dataset Acquisition for a Specific Target Domain}},
  booktitle = {Workshops @ ICCV},
  year = {2023},
  doi = {10.1109/iccvw60793.2023.00438},
}
Figure