Transfer Learning from Synthetic to Real Images Using Variational Autoencoders for Precise Position Detection

Tadanobu Inoue, Subhajit Chaudhury, Giovanni De Magistris, Sakyasingha DasguptaICIP 2018 · DOI

TL;DR

This paper tackles the challenge of accurately detecting object positions in real-world scenarios by bridging the "reality gap" between synthetic and real images. They propose a novel transfer learning method using Variational Autoencoders (VAEs) to generate common "pseudo-synthetic" images, enabling a Convolutional Neural Network (CNN) to learn precise object detection from a limited set of real-world data and a large synthetic dataset.

What Problem Does This Paper Address?

Training deep learning models for tasks like precise object detection typically requires vast amounts of labeled real-world data. Acquiring and annotating such datasets is incredibly time-consuming and expensive. While synthetic data generated in simulations is abundant and easy to create, there's a significant "reality gap" – differences in appearance, lighting, and textures – that often leads to poor performance when models trained solely on synthetic data are applied to real-world scenarios. This paper aims to overcome this gap by effectively transferring knowledge learned from readily available synthetic data to achieve high-accuracy object position detection in the real world, even with minimal real-world training data.

Methods

The core of the proposed method involves a two-stage VAE-based transfer learning approach:

  1. Variational Real-to-Synthetic Mapping:
    • VAE1: This VAE is trained to encode and decode synthetic images, essentially learning a representation of the synthetic domain.
  2. VAE2: This VAE is initialized with VAE1's weights. It's then trained to encode real images and decode them into synthetic-like images. Crucially, the decoder layers are fixed, and only the encoder parameters are adapted. This forces the latent space representations of real and synthetic images to be similar, creating a common "pseudo-synthetic" image space.
  3. Object Detection on Common Image Space:
    • A CNN is trained to detect object positions using the output images from VAE1 (which are synthetic). Because VAE1 and VAE2 share the same decoder, the output image spaces are identical.
  4. During inference, real images are fed into VAE2, which generates blurry pseudo-synthetic images. These images are then passed to the trained CNN to predict the object's position.

This approach leverages the large synthetic dataset for training VAE1 and the CNN, while using a small amount of real data to train VAE2, effectively bridging the domain gap.

Datasets

  • Synthetic Data: Large-scale synthetic images were generated using the Gazebo simulator. These images were used to train VAE1 and subsequently the CNN.
  • Real Data: A limited dataset of real-world images was collected using a Kinect camera. This data was used to train VAE2.
  • Simple Real Objects: Experiments involved five simple 3D-printed objects (cubes, cylinder, triangular prism) with varying colors.
  • Complex Textured Household Objects: Experiments also included five more complex, textured household objects.
  • Benchmark: The paper implicitly establishes a baseline by comparing its method against a CNN trained directly on real images without the VAE-based transfer learning.

Key Insights and Ideas

  • Bridging the Reality Gap with Shared Latent Spaces: The key insight is that by forcing real and synthetic images into a common latent space using VAEs, we can generate "pseudo-synthetic" images that are similar enough for a downstream CNN to perform well, even if the generated images are blurry.
  • Leveraging Abundant Synthetic Data: The method effectively utilizes the vast availability of synthetic data for robust CNN training, mitigating the need for extensive real-world labeling.
  • Robustness to Environmental Variations: The VAE-based approach demonstrates remarkable invariance to changes in lighting conditions, the presence of distractor objects, and different backgrounds, which are common challenges in real-world perception.
  • Selectivity in Distractor Environments: VAE2 can learn to reconstruct a target object even when presented with multiple distractor objects in the scene, demonstrating a useful form of selective attention.

Key Results

  • High Precision: The proposed method achieved an average accuracy of 1.5 mm to 3.5 mm in precise object position detection.
  • Significant Improvement over Baseline: Compared to a baseline CNN trained directly on real images, the VAE-based transfer learning method showed a considerable reduction in prediction errors, particularly for complex textured objects.
  • Robustness Demonstrated: Experimental results showed consistent performance across different lighting conditions (e.g., room light on/off, table light on/off) and in the presence of multiple distractor objects.
  • Real-World Application: The method was successfully demonstrated in a practical robotic "pick-and-place" task.

Limitations and Future Work

  • Blurry Pseudo-Synthetic Images: While the CNN can learn from them, the pseudo-synthetic images generated by VAE2 are described as blurry. This might limit the fine-grained details the CNN can extract compared to perfectly sharp synthetic or real images.
  • Unseen Object Combinations: The paper tested unseen object combinations and backgrounds without retraining, which is a strength. However, the performance on entirely novel object types not seen during VAE training might still be a challenge.
  • Future Work: The authors demonstrated the method's application in a pick-and-place task, suggesting future work could involve more complex robotic manipulation tasks. Further exploration into improving the sharpness of generated images or extending the method to other perception tasks could also be avenues for future research.

Summary updated

BibTeX
@inproceedings{inoue2018transfer,
  author = {Tadanobu Inoue and Subhajit Chaudhury and Giovanni De Magistris and Sakyasingha Dasgupta},
  title = {{Transfer Learning from Synthetic to Real Images Using Variational Autoencoders for Precise Position Detection}},
  booktitle = {ICIP},
  year = {2018},
  doi = {10.1109/icip.2018.8451064},
}
Figure