Abstract
Training deep learning models for semantic segmentation is traditionally labor-intensive, requiring extensive manual annotation of real-world imagery. While synthetic data from simulators offers a scalable alternative with automated labeling, it introduces challenges related to data redundancy and the "reality gap." This paper explores intelligent acquisition strategies to select the most effective synthetic images for training. Our findings demonstrate that selecting a strategic 5% subset of synthetic data can achieve 95.6% of the performance of the full dataset. Furthermore, in adverse real-world conditions, targeted sampling can outperform training on the entire synthetic set by avoiding simulation bias.
Introduction
Semantic segmentation—the task of identifying every pixel in an image—is a cornerstone of autonomous systems. However, annotating a single real-world frame can take up to 90 minutes. Synthetic data from simulators like GTA-5 or CARLA provides a solution, yet it presents two primary obstacles: the "Firehose Problem," where infinite data generation leads to computational redundancy, and the "Reality Gap," where synthetic biases degrade real-world performance. This research investigates whether intelligent sampling can identify the specific synthetic images that best bridge the gap to a target real-world environment.
Methodology
The proposed framework utilizes a simulation engine that populates an image buffer, which is then filtered by an acquisition function to select training samples. Two categories of strategies were evaluated:
- Simulation-only Functions: These prioritize internal data characteristics, such as Class Uniform Sampling (balancing rare objects like traffic lights) and Entropy/CoreSet (selecting images based on model uncertainty or set diversity).
- Correspondence-based Functions: This novel approach identifies synthetic "twins" of real-world "corner cases"—scenarios where the model currently struggles in the target domain. By using Cosine Similarity in the feature space, the system selects synthetic data that directly addresses real-world failures.
Experimental Setup
The study utilized three primary datasets for "Synthetic-to-Real" adaptation:
- GTA-5 (Source): 25,000 synthetic frames serving as the simulation environment.
- Cityscapes (Target): Real-world European street scenes under favorable weather conditions.
- ACDC (Target): A challenging dataset featuring adverse conditions such as fog, rain, snow, and nighttime.
Key Results and Discussion
The Efficiency of Intelligent Sampling
On the Cityscapes dataset, the intelligent acquisition strategy achieved a Mean Intersection over Union (mIoU) of 66.17% using only 5% of the available synthetic data (approximately 1,250 frames). This represents 95.6% of the performance achieved by the full dataset, highlighting the significant redundancy in raw simulation output.
The "Less is More" Paradox
In tests involving the ACDC dataset, training on 100% of the synthetic data resulted in lower accuracy than training on a targeted 5% subset. Specifically, the "uni CC max sim" strategy—which combines class balancing with real-world corner-case matching—achieved an mIoU of 49.3%, surpassing the 46.37% achieved by the full dataset. This suggests that large-scale synthetic data can contain "toxic" information or biases that confuse models when applied to complex, adverse real-world scenes.
Conclusion and Future Work
Intelligent synthetic data acquisition effectively mitigates the computational costs of simulation while improving model robustness in challenging environments. However, the success of correspondence-based sampling remains dependent on the quality of the feature encoder. Future research will focus on exploring alternative feature space embeddings and validating these acquisition functions across a broader range of diverse real-world camera systems and environmental conditions.
Summary updated