Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects

Jonathan Tremblay, Thang To, Balakumar Sundaralingam, Yu Xiang, Dieter Fox, Stan Birchfield

Abstract

We present Deep Object Pose Estimation (DOPE), a method for detecting and estimating the 6-DoF pose of known household objects from a single RGB image. Our approach addresses the challenge of the "reality gap" by training a deep neural network entirely on synthetic data. By combining domain randomization with high-fidelity photorealistic simulations, we demonstrate that a model can generalize to real-world environments without requiring any real-world training images. Our system achieves state-of-the-art results on standard benchmarks and enables a robot to successfully grasp objects in cluttered, real-world scenes.

Introduction

For robots to interact effectively with human environments, they must possess a precise understanding of the 3D position and orientation (6-DoF pose) of objects. While deep learning has revolutionized 2D object detection, 3D pose estimation remains a significant challenge due to the difficulty of obtaining large-scale, accurately labeled real-world datasets. Synthetic data offers a solution by providing infinite, perfectly labeled training examples; however, models trained solely on synthetic data often fail when deployed in the real world—a phenomenon known as the reality gap.

This paper introduces DOPE, a system designed to bridge this gap. By leveraging a novel combination of synthetic training strategies, DOPE achieves robust performance in the real world without ever seeing a real training image during the learning process.

The DOPE Framework

DOPE utilizes a multi-stage fully convolutional network inspired by Convolutional Pose Machines. The network takes a single RGB image as input and produces two primary outputs:

  1. Belief Maps: These maps predict the 2D locations of nine keypoints for each object: the eight vertices of its 3D bounding box and its centroid.
  2. Vector Fields: These fields help the system associate specific vertices with the correct centroid, enabling the detection of multiple instances of the same object in a single frame.

Once the 2D keypoints are identified, the 6-DoF pose is recovered using a standard Perspective-n-Point (PnP) algorithm. This geometric approach allows the network to remain independent of specific camera intrinsics, enhancing its generalizability across different hardware.

Training Data: Bridging the Reality Gap

The core innovation of DOPE lies in its training data strategy. Rather than relying on a single type of simulation, the authors combine two complementary approaches:

  • Domain Randomization (DR): This involves placing objects against random backgrounds with non-realistic lighting and textures. This forces the network to focus on the essential geometric features of the objects rather than overfitting to specific environmental details.
  • Photorealistic Simulation: Using a custom plugin for Unreal Engine 4, the authors generated high-fidelity images that include realistic physics, lighting, and complex occlusions.
Comparison of Domain Randomized (left) and Photorealistic (right) synthetic training data used to train the DOPE network.

The combination of these two datasets ensures the model is both robust to environmental variations and capable of handling the nuances of real-world lighting and object interactions.

Experimental Results

DOPE was evaluated on the YCB-Video dataset, a standard real-world benchmark. The results indicate that the combined "DR+photo" training strategy consistently outperforms models trained on either data type alone. Furthermore, DOPE achieved performance comparable to or better than PoseCNN, a leading model that incorporates real-world training data.

ADD pass rate curves for five household objects, demonstrating that the combined DR+photo approach (blue line) consistently achieves high accuracy compared to single-strategy models and the PoseCNN baseline.

In real-world robotic trials using a Baxter robot, DOPE provided poses accurate enough for successful grasping in cluttered environments. The system demonstrated high success rates for objects like sugar boxes and mustard bottles (11/12 successful grasps), though it faced challenges with highly reflective surfaces, such as the metallic tops of soup cans.

Conclusion

DOPE demonstrates that the reality gap can be bridged using only synthetic data by strategically mixing domain randomization with photorealism. This approach eliminates the need for expensive real-world data labeling and results in a robust, real-time pose estimation system. Future work will focus on improving material modeling for reflective surfaces and refining grasping behavior through closed-loop control.

Summary updated

Abstract

Using synthetic data for training deep neural networks for robotic manipulation holds the promise of an almost unlimited amount of pre-labeled training data, generated safely out of harm’s way. One...
BibTeX
@inproceedings{tremblay2018deep,
  author = {Jonathan Tremblay and Thang To and Balakumar Sundaralingam and Yu Xiang and Dieter Fox and Stan Birchfield},
  title = {{Deep Object Pose Estimation for Semantic Robotic Grasping of Household Objects}},
  booktitle = {Conference on Robot Learning},
  year = {2018},
  url = {https://openreview.net/forum?id=fXZTAweuxH},
}
Figure