Abstract
Vision and learning have made significant progress that could improve robotics policies for complex tasks and environments. Learning deep neural networks for image understanding, however, requires large amounts of domain-specific visual data. While collecting such data from real robots is possible, such an approach limits the scalability as learning policies typically requires thousands of trials. In this work we attempt to learn manipulation policies in simulated environments. Simulators enable scalability and provide access to the underlying world state during training. Policies learned in simulators, however, do not transfer well to real scenes given the domain gap between real and synthetic data. We follow recent work on domain randomization and augment synthetic images with sequences of random transformations. Our main contribution is to optimize the augmentation strategy for sim2real transfer and to enable domain-independent policy learning. We design an efficient search for depth image augmentations using object localization as a proxy task. Given the resulting sequence of random transformations, we use it to augment synthetic depth images during policy learning. Our augmentation strategy is policy-independent and enables policy learning with no real images. We demonstrate our approach to significantly improve accuracy on three manipulation tasks evaluated on a real robot.
Introduction: The Reality Gap
Training robots in the real world is a slow, expensive, and often dangerous process. While physics simulators offer a safe and scalable alternative, they suffer from the "reality gap": a robot policy that masters a task in simulation often fails when faced with the messy, noisy visual data of the real world.
Domain Randomization (DR) is a popular fix where researchers manually add noise, change textures, or jitter lighting to force the robot's neural network to ignore simulation-specific artifacts. However, manual DR is often a "black art" requiring tedious trial and error. This work proposes to automate the search for the optimal domain randomization strategy.
Methodology
The proposed approach breaks the problem into a three-stage pipeline: learning an augmentation function, training control policies using that function, and finally running those policies in the real world.
1. The Proxy Task
Training a full robot policy is computationally expensive. To speed up the search for augmentations, the authors used a "proxy task": predicting the 3D position of a cube from a depth image. The core assumption is that if an augmentation helps a network locate an object in a real image, it will also help a robot policy interact with that object.
2. Monte Carlo Tree Search (MCTS)
The researchers defined 11 primitive transformations, including standard ones (Scale, Sharpness, White Noise) and "object-aware" ones (Boundary Noise, Erasing Objects) that utilize the simulator's segmentation masks. To find the best sequence of 8 transformations out of a massive search space ($\\approx 3.6 \\times 10^{14}$ combinations), they employed MCTS. The algorithm samples a sequence, trains the proxy task on augmented synthetic data, and evaluates it on a small set of 200 real-world depth images. The error on these real images guides the search.
3. Policy Learning
Once the optimal augmentation sequence is found, it is applied to synthetic data to train a robot policy using Behavior Cloning (BC). The robot learns to mimic an expert script while seeing only these "corrupted" synthetic images, ensuring it is robust to the noise it will encounter in reality.
Key Insights
- Proxy Tasks are Powerful: A simple localization task is a surprisingly effective stand-in for complex manipulation when searching for visual augmentations.
- The "Reality Gap" Trade-off: Models that performed better on real data often performed worse on clean synthetic data, suggesting the augmentations successfully force the model to ignore "easy" simulation features that don't exist in reality.
- Depth-Specific Noise: Standard RGB augmentations are insufficient for depth sensors. Transformations like "BoundaryNoise," which mimics the fuzzy edges of real depth sensors, were crucial for success.
Experimental Results
The learned augmentation strategy significantly outperformed both "None" (no augmentation) and handcrafted baselines across three manipulation tasks evaluated on a real robot:
| Task | No Augmentation | Handcrafted Aug. | Learned Aug. (Ours) |
|---|---|---|---|
| Cube Picking | 15% | 45% | 95% |
| Cube Stacking | 5% | 10% | 90% |
| Cup Placing | 0% | 30% | 75% |
In the Cube Picking task, the robot succeeded in 19 out of 20 trials. Even in the difficult Cup Placing task—which involved objects the robot had never seen before—the learned augmentations allowed for a 75% success rate, whereas the model trained on standard simulation data failed every single time.
Conclusion and Future Work
This work demonstrates that automated augmentation search can effectively bridge the Sim2Real gap for depth-based manipulation. While the policy training is zero-shot, the MCTS search currently requires a small set of labeled real-world images. Future research could explore dynamic proxy tasks to capture temporal noise or extend the framework to full RGB data, which introduces additional complexities like lighting and shadows.
Summary updated