Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics

Jeffrey Mahler, Jacky Liang, Sherdil Niyaz, Michael Laskey, Richard Doan, Xinyu Liu, Juan Aparicio Ojea, Ken Goldberg

Introduction

Reliable robotic grasping remains a significant challenge due to inherent uncertainties in sensor data, object geometry, and physical properties like friction. Traditional approaches typically fall into two categories: analytic methods, which rely on precise physics but fail under real-world noise, and empirical methods, which require massive amounts of time-consuming physical data collection.

Dex-Net 2.0 bridges this gap by training a Grasp Quality Convolutional Neural Network (GQ-CNN) entirely on 6.7 million synthetic data points. This approach overcomes the "data bottleneck" by using physics-based simulations to generate a dataset large and diverse enough to generalize to novel, complex objects in the real world.

Dex-Net 2.0 Pipeline: From depth image input to GQ-CNN ranking and grasp execution.

Methods

The core of Dex-Net 2.0 is a pipeline that translates analytic grasp theory into a deep learning framework.

Synthetic Data Generation

The researchers utilized 1,500 3D models to simulate millions of grasp attempts. Each attempt was evaluated using "Robust Epsilon Quality," a metric measuring a grasp's resistance to external forces while accounting for uncertainty in gripper pose and object friction. To bridge the reality gap, they employed a stochastic observation model that renders 2.5D point clouds (depth images) with realistic Gaussian and Gamma noise, mimicking standard depth cameras.

Probabilistic graphical model and physical parameters used for synthetic data generation.

The GQ-CNN Architecture

The GQ-CNN is designed to predict the probability of grasp success. It takes a depth image patch—centered and rotated to align with a candidate grasp—along with the gripper's depth relative to the camera. This "alignment trick" removes the need for the network to learn rotational invariance, significantly increasing learning efficiency.

Grasp Planning Policy

During execution, the robot samples hundreds of antipodal grasp candidates. The GQ-CNN ranks these candidates based on predicted robustness, and the robot executes the highest-ranked grasp. This process takes approximately 0.8 seconds, enabling near real-time planning.

Datasets

The study utilized three primary datasets:

  • Dex-Net 2.0: 6.7 million synthetic point clouds and grasps derived from 1,500 3D models.
  • Adv-Synth: A validation set of 8 3D-printed objects with adversarial geometries like smooth curves.
  • Novel Household Objects: 40 real-world items, including articulated and deformable objects, used to evaluate generalization.

Key Results and Insights

The results demonstrate that models trained exclusively on synthetic data can achieve high performance on physical hardware:

  • Precision and Success: The system achieved 99% precision on novel household objects and a 93% success rate on adversarial geometries.
  • Robustness over Success: Training the network to predict a robustness metric (factoring in uncertainty) rather than binary success proved critical for real-world reliability.
  • Generalization: The model successfully grasped objects it had never encountered, including articulated items like pliers and deformable items like washcloths.

Limitations and Future Work

While highly effective for singulated objects, Dex-Net 2.0 currently struggles with cluttered environments where occlusions and collisions are frequent. It also faces challenges with very thin objects, such as spatulas, which exceed the resolution of current depth sensors. Future work aims to extend the system to handle clutter, incorporate multi-view vision, and utilize active learning to improve model performance through exploration.

Summary updated

BibTeX
@inproceedings{mahler2017dex,
  author = {Jeffrey Mahler and Jacky Liang and Sherdil Niyaz and Michael Laskey and Richard Doan and Xinyu Liu and Juan Aparicio Ojea and Ken Goldberg},
  title = {{Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metrics}},
  booktitle = {Robotics: Science and Systems},
  year = {2017},
  doi = {10.15607/rss.2017.xiii.058},
}
Figure