Abstract
We present Deep Object Pose Estimation (DOPE), a method for detecting and estimating the 6-DoF pose of known household objects from a single RGB image. Our approach addresses the challenge of the "reality gap" by training a deep neural network entirely on synthetic data. By combining domain randomization with high-fidelity photorealistic simulations, we demonstrate that a model can generalize to real-world environments without requiring any real-world training images. Our system achieves state-of-the-art results on standard benchmarks and enables a robot to successfully grasp objects in cluttered, real-world scenes.
Introduction
For robots to interact effectively with human environments, they must possess a precise understanding of the 3D position and orientation (6-DoF pose) of objects. While deep learning has revolutionized 2D object detection, 3D pose estimation remains a significant challenge due to the difficulty of obtaining large-scale, accurately labeled real-world datasets. Synthetic data offers a solution by providing infinite, perfectly labeled training examples; however, models trained solely on synthetic data often fail when deployed in the real world—a phenomenon known as the reality gap.
This paper introduces DOPE, a system designed to bridge this gap. By leveraging a novel combination of synthetic training strategies, DOPE achieves robust performance in the real world without ever seeing a real training image during the learning process.
The DOPE Framework
DOPE utilizes a multi-stage fully convolutional network inspired by Convolutional Pose Machines. The network takes a single RGB image as input and produces two primary outputs:
- Belief Maps: These maps predict the 2D locations of nine keypoints for each object: the eight vertices of its 3D bounding box and its centroid.
- Vector Fields: These fields help the system associate specific vertices with the correct centroid, enabling the detection of multiple instances of the same object in a single frame.
Once the 2D keypoints are identified, the 6-DoF pose is recovered using a standard Perspective-n-Point (PnP) algorithm. This geometric approach allows the network to remain independent of specific camera intrinsics, enhancing its generalizability across different hardware.
Training Data: Bridging the Reality Gap
The core innovation of DOPE lies in its training data strategy. Rather than relying on a single type of simulation, the authors combine two complementary approaches:
- Domain Randomization (DR): This involves placing objects against random backgrounds with non-realistic lighting and textures. This forces the network to focus on the essential geometric features of the objects rather than overfitting to specific environmental details.
- Photorealistic Simulation: Using a custom plugin for Unreal Engine 4, the authors generated high-fidelity images that include realistic physics, lighting, and complex occlusions.
The combination of these two datasets ensures the model is both robust to environmental variations and capable of handling the nuances of real-world lighting and object interactions.
Experimental Results
DOPE was evaluated on the YCB-Video dataset, a standard real-world benchmark. The results indicate that the combined "DR+photo" training strategy consistently outperforms models trained on either data type alone. Furthermore, DOPE achieved performance comparable to or better than PoseCNN, a leading model that incorporates real-world training data.
In real-world robotic trials using a Baxter robot, DOPE provided poses accurate enough for successful grasping in cluttered environments. The system demonstrated high success rates for objects like sugar boxes and mustard bottles (11/12 successful grasps), though it faced challenges with highly reflective surfaces, such as the metallic tops of soup cans.
Conclusion
DOPE demonstrates that the reality gap can be bridged using only synthetic data by strategically mixing domain randomization with photorealism. This approach eliminates the need for expensive real-world data labeling and results in a robust, real-time pose estimation system. Future work will focus on improving material modeling for reflective surfaces and refining grasping behavior through closed-loop control.
Summary updated