Abstract: It is very time consuming to create datasets for training computer vision models. An emerging alternative is to use synthetic data, but if the synthetic data is not similar enough to the real data, the performance is typically below that of training with real data. Thus using synthetic data still requires a large amount of time, money, and skill as one needs to author the data carefully. In this paper, we seek to understand which aspects of this authoring process are most critical. We present an analysis of which factors of variation between simulated and real data are most important. We capture images of YCB objects to create a novel YCB-Real dataset. We then create a novel synthetic "digital twin" dataset, YCB-Synthetic, which matches the YCB-Real dataset and includes variety of artifacts added to the synthetic data. We study the affects of these artifacts on our dataset and two existing published datasets on two different computer vision tasks: object detection and instance segmentation. We provide an analysis of the cost-benefit trade-offs between artist time for fixing artifacts and trained model accuracy. We plan to release this dataset (images and 3D assets) so they can be further used by the community.
Introduction
Training robust computer vision models requires massive, accurately labeled datasets. While synthetic data—rendered from 3D models—offers a scalable alternative to manual labeling, models trained on "fake" data often suffer from a performance drop when deployed in the real world. This phenomenon, known as the Sim2Real gap, is typically addressed by employing technical artists to manually refine 3D assets. Artists fix textures, close mesh holes, and adjust lighting to make the simulation as realistic as possible.
However, this manual "cleaning" process is expensive and time-consuming. Until now, there has been little empirical data to guide artists on which specific 3D artifacts actually degrade model performance and which are negligible. This research seeks to identify the most critical factors in synthetic data authoring to optimize the "bang for your buck" in artist labor.
Methodology and Datasets
The researchers developed a controlled experimental framework using the concept of "Digital Twins." By matching synthetic environments to real-world captures, they could isolate the impact of specific modeling errors.
Asset and Dataset Creation
- YCB-Real (New): The team selected 20 items from the Yale-Carnegie Mellon-Berkeley (YCB) object set and captured ~100 images per object in a controlled lab setting with known lighting and camera positions.
- YCB-Synthetic (New): A matching digital twin dataset was created. Rather than aiming for perfection, the researchers systematically injected five common photogrammetry artifacts at varying severity levels: mesh noise, geometric holes, texture blur, ambient lighting shifts, and baked lighting.
- Benchmark Datasets: To ensure findings generalized beyond controlled settings, the researchers also evaluated models on the existing YCB-In-the-wild and YCB-Video datasets, which feature more complex and cluttered environments.
Evaluation Framework
Faster R-CNN (for detection) and Mask R-CNN (for segmentation) models were trained on these "corrupted" synthetic datasets. Professional 3D artists provided time estimates for fixing each artifact type, allowing for a direct comparison between labor hours and mean Average Precision (mAP) recovery.
Key Insights and Results
The Impact of Artifacts
- Mesh Noise is the "Silent Killer": High-frequency noise in the 3D mesh caused the most significant performance drops. Interestingly, this noise primarily led to classification confusions (e.g., mistaking a pitcher for a mug) rather than localization failures. Training on clean data yielded ~82% mAP, but Level 3 mesh noise plummeted that score to 65.6%.
- Lighting Resilience: Changes in ambient lighting had almost zero impact on performance. Modern neural network architectures, likely due to components like Batch Normalization, appear robust to global lighting shifts.
- Baked Lighting: When shadows or highlights are "painted" onto a texture (baked) rather than calculated dynamically, performance dropped by nearly 10 mAP points. This suggests that physically inaccurate lighting is highly detrimental to model generalization.
- Data Augmentation Effects: Surprisingly, some artifacts like minor geometric holes or texture blur occasionally boosted performance for specific objects, acting as a form of unintended data augmentation that forced the model to learn more robust features.
Cost-Benefit Analysis
The study revealed a clear hierarchy of artist priorities. Fixing baked lighting and significant mesh noise provides the highest return on investment. Conversely, spending hours closing small holes in a mesh or fine-tuning ambient light is often a waste of resources.
As shown in the analysis, fixing ambient light takes approximately 4 hours but yields nearly 0 mAP gain. In contrast, addressing Level 3 mesh noise takes roughly 7 hours but can recover over 15 mAP points, representing a far more efficient use of technical artistry.
Conclusion
This research systematically decomposes the Sim2Real gap, providing a roadmap for efficient synthetic data generation. By focusing on mesh integrity and dynamic lighting over minor geometric perfections, developers can significantly reduce the cost of training high-performance vision models.
Future work will explore how these sensitivities vary with object size and whether these trends hold in different domains, such as autonomous driving or large-scale architectural scenes, where the Sim2Real gap remains a primary hurdle.
Summary updated