Close the Sim2real Gap via Physically-based Structured Light Synthetic Data Simulation

Kaixin Bai, Lei Zhang, Zhaopeng Chen, Fang Wan, Jianwei Zhang

Introduction

In industrial robotics, deep learning offers significant perception capabilities but is often bottlenecked by the need for massive labeled datasets. Collecting and hand-labeling thousands of images of metal parts in factory settings is prohibitively expensive and slow. While synthetic data (sim2real) is a promising alternative, it frequently fails because synthetic images are often too "perfect," lacking the specific noise and artifacts present in real-world sensor data. This paper introduces a physically-based simulation pipeline that bridges this gap by simulating the actual physics of structured light cameras.

Methodology: Simulating the Sensing Process

The core innovation lies in simulating the process of sensing rather than just the scene. Built on the Blender Cycles ray-tracing engine, the pipeline models the entire structured light acquisition sequence:

  1. Pattern Projection: A virtual projector casts Gray Code patterns onto the 3D scene.
  2. Physically-Based Rendering: Ray tracing and AI denoising (NVIDIA OptiX) simulate light interaction with complex materials, such as reflective metal.
  3. Algorithmic Reconstruction: Simulated 2D pattern images are processed through the same 3D reconstruction algorithms used by physical cameras.
The proposed simulation and physical world pipeline for bridging the sim2real gap.

By accounting for light paths and shadows, the resulting depth maps naturally incorporate "decoding errors" and "flying noise" characteristic of real-world data, making the synthetic data indistinguishable from real sensor feeds to neural networks.

Experimental Setup and Datasets

To validate the approach, the researchers generated 1,000 synthetic scenes for five object classes, including industrial metal parts and household items from the KIT object models database. Performance was evaluated against a real-world test set of 100 captures per object and a baseline dataset generated using standard domain randomization in NVIDIA Isaac Sim.

Key Insights

  • Sensor-Centric Simulation: Realistic noise is a structured byproduct of physics and reconstruction math. Simulating the sensing process provides this noise naturally.
  • Depth Robustness: Depth images are less sensitive to environmental lighting changes than RGB. Accurate depth noise simulation allows for better generalization to real-world environments.
  • Efficiency via AI Denoising: Using AI denoisers significantly accelerates high-fidelity data generation, reducing development time from weeks to days.

Results and Real-World Application

The proposed method demonstrated a minimal sim2real gap. For specific industrial workpieces, the performance gap between synthetic and real data was reduced to 0.017, compared to 0.077 for RGB-based models. In object detection tasks (YOLOv7), the method achieved an IoU (Intersection over Union) of 0.648 on real data, outperforming standard domain-randomized models (0.584).

Robotic grasping system utilizing the sim2real perception model for bin-picking.

The practical utility was confirmed in robotic bin-picking tasks using UR5e and Diana7 robots, where the integration of this perception model increased grasping success rates from 95.6% to 98.0%.

Conclusion and Future Work

This research successfully demonstrates that physically-based sensor simulation can effectively close the sim2real gap for industrial applications. Future work will focus on scaling the dataset to include more diverse objects, refining pose estimation algorithms, and addressing the challenges posed by extreme materials like highly transparent or mirror-like surfaces.

Summary updated

BibTeX
@inproceedings{bai2024close,
  author = {Kaixin Bai and Lei Zhang and Zhaopeng Chen and Fang Wan and Jianwei Zhang},
  title = {{Close the Sim2real Gap via Physically-based Structured Light Synthetic Data Simulation}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2024},
  doi = {10.1109/icra57147.2024.10611401},
}
Figure