Introduction
In industrial robotics, deep learning offers significant perception capabilities but is often bottlenecked by the need for massive labeled datasets. Collecting and hand-labeling thousands of images of metal parts in factory settings is prohibitively expensive and slow. While synthetic data (sim2real) is a promising alternative, it frequently fails because synthetic images are often too "perfect," lacking the specific noise and artifacts present in real-world sensor data. This paper introduces a physically-based simulation pipeline that bridges this gap by simulating the actual physics of structured light cameras.
Methodology: Simulating the Sensing Process
The core innovation lies in simulating the process of sensing rather than just the scene. Built on the Blender Cycles ray-tracing engine, the pipeline models the entire structured light acquisition sequence:
- Pattern Projection: A virtual projector casts Gray Code patterns onto the 3D scene.
- Physically-Based Rendering: Ray tracing and AI denoising (NVIDIA OptiX) simulate light interaction with complex materials, such as reflective metal.
- Algorithmic Reconstruction: Simulated 2D pattern images are processed through the same 3D reconstruction algorithms used by physical cameras.
By accounting for light paths and shadows, the resulting depth maps naturally incorporate "decoding errors" and "flying noise" characteristic of real-world data, making the synthetic data indistinguishable from real sensor feeds to neural networks.
Experimental Setup and Datasets
To validate the approach, the researchers generated 1,000 synthetic scenes for five object classes, including industrial metal parts and household items from the KIT object models database. Performance was evaluated against a real-world test set of 100 captures per object and a baseline dataset generated using standard domain randomization in NVIDIA Isaac Sim.
Key Insights
- Sensor-Centric Simulation: Realistic noise is a structured byproduct of physics and reconstruction math. Simulating the sensing process provides this noise naturally.
- Depth Robustness: Depth images are less sensitive to environmental lighting changes than RGB. Accurate depth noise simulation allows for better generalization to real-world environments.
- Efficiency via AI Denoising: Using AI denoisers significantly accelerates high-fidelity data generation, reducing development time from weeks to days.
Results and Real-World Application
The proposed method demonstrated a minimal sim2real gap. For specific industrial workpieces, the performance gap between synthetic and real data was reduced to 0.017, compared to 0.077 for RGB-based models. In object detection tasks (YOLOv7), the method achieved an IoU (Intersection over Union) of 0.648 on real data, outperforming standard domain-randomized models (0.584).
The practical utility was confirmed in robotic bin-picking tasks using UR5e and Diana7 robots, where the integration of this perception model increased grasping success rates from 95.6% to 98.0%.
Conclusion and Future Work
This research successfully demonstrates that physically-based sensor simulation can effectively close the sim2real gap for industrial applications. Future work will focus on scaling the dataset to include more diverse objects, refining pose estimation algorithms, and addressing the challenges posed by extreme materials like highly transparent or mirror-like surfaces.
Summary updated