Learning to Augment Synthetic Images for Sim2Real Policy Transfer

Alexander Pashevich, Robin Strudel, Igor Kalevatykh, Ivan Laptev, Cordelia Schmid

Training robots in the real world is a slow, expensive, and often dangerous process. While physics simulators offer a safe and scalable alternative, they suffer from the "reality gap": a robot policy that masters a task in simulation often fails miserably when faced with the messy, noisy visual data of the real world.

In this paper, researchers from Inria and University Grenoble Alpes propose a novel way to bridge this gap by learning how to make synthetic images look "real enough" for a robot's brain.

TL;DR

The authors present a method to automatically discover the best sequence of image augmentations (like noise and distortions) to bridge the sim-to-real gap. By using a fast "proxy task" and Monte Carlo Tree Search, they learn an augmentation strategy in simulation that allows a robot to perform complex manipulation tasks in the real world with high success rates, using zero real-world images for the actual policy training.

What Problem Does This Paper Address?

The core challenge is Sim2Real transfer. Simulators provide perfect data and "ground truth" (knowing exactly where every object is), but their visual output is too clean. Domain Randomization (DR) is a popular fix where researchers manually add noise, change textures, or jitter lighting to force the robot's neural network to ignore simulation-specific artifacts.

However, manual DR is often a "black art"—it requires tedious trial and error to find which augmentations actually help. This paper asks: Can we automate the search for the optimal domain randomization strategy?

Methods

The researchers broke the problem into two main parts: finding the right augmentations and then training the robot.

  1. The Proxy Task: Training a full robot policy is computationally expensive. To speed up the search for augmentations, the authors used a "proxy task": predicting the 3D position of a cube from a depth image. They assumed that if an augmentation helps a network locate an object in a real image, it will also help a robot policy interact with that object.
  2. The Search Space: They defined 11 primitive transformations, including standard ones (Scale, Sharpness, White Noise) and "object-aware" ones (Boundary Noise, Erasing Objects) that utilize the simulator's segmentation masks.
  3. Monte Carlo Tree Search (MCTS): To find the best sequence of 8 transformations out of a massive search space ($\approx 3.6 \times 10^{14}$ combinations), they employed MCTS. The algorithm samples a sequence, trains the proxy task on augmented synthetic data, and evaluates it on a small set of 200 real-world depth images. The error on these real images guides the search.
  4. Policy Learning: Once the optimal augmentation sequence is found, they apply it to synthetic data to train a robot policy using Behavior Cloning (BC). The robot learns to mimic an expert script while seeing only these "corrupted" synthetic images.

Datasets

  • Simulation: A custom environment built in the pybullet physics engine.
  • Objects: 3D models of cups and plates were sourced from ModelNet and ShapeNet to ensure the robot could generalize to different shapes.
  • Real World: A small dataset of 200 real depth images (captured via a Kinect-1) was used only to evaluate the proxy task during the MCTS search. No real images were used to train the final control policy.

Key Insights and Ideas

  • Proxy Tasks are Powerful: You don't need to simulate the entire robot's behavior to find good visual augmentations. A simple localization task is a surprisingly effective "stand-in" for complex manipulation.
  • The "Reality Gap" Trade-off: The authors found that the better a model performed on real data, the worse it often performed on the original, clean synthetic data. This suggests that the augmentations successfully force the model to ignore "easy" simulation features that don't exist in reality.
  • Depth-Specific Noise Matters: Standard image augmentations designed for RGB photos (like color jittering) aren't enough for depth sensors. Transformations like "BoundaryNoise" (which mimics the fuzzy edges of real depth sensors) were crucial for success.

Key Results

The learned augmentation strategy significantly outperformed "None" (no augmentation) and handcrafted baselines across three tasks:

Task No Augmentation Handcrafted Aug. Learned Aug. (Ours)
Cube Picking 15% 45% 95%
Cube Stacking 5% 10% 90%
Cup Placing 0% 30% 75%

In the Cube Picking task, the robot succeeded in 19 out of 20 trials. Even in the difficult Cup Placing task—which involved objects the robot had never seen before—the learned augmentations allowed for a 75% success rate, whereas the model trained on standard simulation data failed every single time.

Limitations and Future Work

  • Requirement for Real Data: While the policy training is "zero-shot" (no real images), the MCTS search still requires a small set of labeled real-world images for the proxy task.
  • Computational Cost: Searching for the optimal augmentation sequence took about 12 hours on 16 GPUs. While this is a one-time cost, it is still significant.
  • Static Scenes: The current proxy task focuses on static object localization. Future work could explore dynamic proxy tasks to better capture motion blur or temporal noise in video streams.
  • RGB Extension: The paper focuses on depth images. Extending this to full RGB data would introduce more complexity (lighting, shadows, textures) but could further broaden the method's applicability.

Summary updated

Abstract

Vision and learning have made significant progress that could improve robotics policies for complex tasks and environments. Learning deep neural networks for image understanding, however, requires large amounts of domain-specific visual data. While collecting such data from real robots is possible, such an approach limits the scalability as learning policies typically requires thousands of trials. In this work we attempt to learn manipulation policies in simulated environments. Simulators enable scalability and provide access to the underlying world state during training. Policies learned in simulators, however, do not transfer well to real scenes given the domain gap between real and synthetic data. We follow recent work on domain randomization and augment synthetic images with sequences of random transformations. Our main contribution is to optimize the augmentation strategy for sim2real transfer and to enable domain-independent policy learning. We design an efficient search for depth image augmentations using object localization as a proxy task. Given the resulting sequence of random transformations, we use it to augment synthetic depth images during policy learning. Our augmentation strategy is policy-independent and enables policy learning with no real images. We demonstrate our approach to significantly improve accuracy on three manipulation tasks evaluated on a real robot.
BibTeX
@inproceedings{pashevich2019learning,
  author = {Alexander Pashevich and Robin Strudel and Igor Kalevatykh and Ivan Laptev and Cordelia Schmid},
  title = {{Learning to Augment Synthetic Images for Sim2Real Policy Transfer}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=rMyl2BoKy62J},
}

Loading related content...

Figure