DVGG: Deep Variational Grasp Generation for Dextrous Manipulation

Wei Wei, Daheng Li, Peng Wang, Yiming Li, Wanyi Li, Yongkang Luo, Jun Zhong

TL;DR

This paper introduces DVGG, a deep learning model that generates diverse and physically plausible grasp configurations for anthropomorphic robotic hands from a single image. By combining point cloud completion, a variational grasp generator, and an iterative refinement module, DVGG achieves a significant improvement in grasp success rates for novel objects on real robotic platforms.

What Problem Does This Paper Address?

Grasping with anthropomorphic robotic hands, which have multiple joints and degrees of freedom, is significantly more complex than with simple parallel-jaw grippers. Traditional methods often require known object models and can be computationally expensive, making them unsuitable for unknown objects in real-world scenarios. While deep learning has shown promise, most existing approaches focus on simpler grippers or struggle with the high dimensionality of multi-finger hand configurations. The challenge lies in generating a variety of high-quality, physically plausible grasps for unseen objects using only a single sensor view.

Methods

DVGG employs a three-stage generative pipeline:

  1. Point Cloud Completion: Given a partial point cloud observation of an object, this module reconstructs a complete 3D point cloud. This is crucial because many grasp generation methods assume complete object knowledge. The authors adapt a method from [40] and make modifications for easier real-world application.
  2. Variational Grasp Generator (VGG): This module uses a Conditional Variational Auto-Encoder (CVAE) to generate diverse sets of "coarse" grasp configurations. It takes the complete object point cloud as input and learns to map it to a latent space from which grasp poses can be sampled. The VGG is trained to reconstruct hand meshes and predict hand joint configurations, incorporating hand-object interaction constraints.
  3. Iterative Grasp Refinement: To address issues like penetration and loose contacts, this module refines the coarse grasps generated by the VGG. It iteratively adjusts the grasp pose by penalizing hand-object interpenetration and optimizing contact energy, aiming for more stable and physically plausible grasps.

Datasets

  • Training Data: The authors created a large-scale synthetic grasping dataset containing approximately 300 common object models with 1.5 million annotated grasps. This dataset was generated in a physics simulator.
  • Evaluation Data:
    • Simulation: Experiments were conducted on objects from the YCB dataset (seen) and EGAD! dataset (unseen).
  • Real Robot Platform: Experiments were performed using objects from the YCB dataset (similar to training) and novel objects.

Key Insights and Ideas

  • Contact-based Grasp Generation: The core idea is to model grasps based on implicit hand-object contacts, which is more intuitive and robust for anthropomorphic hands.
  • Coarse-to-Fine Generation: The two-stage approach (VGG for initial diverse grasps, then refinement) effectively balances generating a wide variety of grasps with ensuring their physical plausibility.
  • Importance of Point Cloud Completion: For single-view grasping of unknown objects, accurately completing the object's 3D model is a critical prerequisite.
  • VAE for Diversity and Speed: Using a Variational Auto-Encoder allows for efficient generation of diverse grasp configurations, significantly outperforming traditional methods like GraspIt! in terms of speed.
  • Iterative Refinement for Robustness: The refinement module is essential for improving grasp quality by actively addressing penetration and contact tightness, leading to higher success rates in real-world scenarios.

Key Results

  • Simulation: DVGG achieved significantly higher grasp success rates compared to baselines like GraspIt! on both seen and unseen objects. For example, on the EGAD! dataset, DVGG achieved a success rate of 74.9% (with refinement) compared to GraspIt!'s 27.0%.
  • Real Robot Platform: DVGG demonstrated strong performance on a real robotic platform. For novel objects, DVGG with refinement achieved a grasp success rate of 70.7%, a substantial improvement over GraspIt! (46.0%) and PointNetGPD (48.3%).
  • Efficiency: DVGG generates grasps in approximately 0.16 seconds, which is orders of magnitude faster than GraspIt! (35+ seconds).
  • Ablation Studies: The paper highlights the importance of each component: removing the penetration loss significantly degraded performance, and the hand-object contact loss was crucial for success rate.

Limitations and Future Work

  • Failure Cases: The paper acknowledges some failure cases on the real robot platform, primarily due to unstable grasps requiring large frictional forces or collisions between the hand and the object during approach.
  • Friction Dependence: The success of some grasps is shown to be dependent on friction, suggesting that future work could explore more robust strategies for low-friction scenarios.
  • Generalization to Clutter: While the paper focuses on single objects, future work could extend the approach to handle grasping in cluttered environments.
  • Dataset Size and Diversity: While the dataset is large, further expansion with more diverse object types and grasping scenarios could improve generalization.

Summary updated

Abstract

Grasping with anthropomorphic robotic hands involves much more hand-object interactions compared to parallel-jaw grippers. Modeling hand-object interactions is essential to the study of multi-finger hand dextrous manipulation. This work presents DVGG, an efficient grasp generation network that takes single-view observation as input and predicts high-quality grasp configurations for unknown objects. In general, our generative model consists of three components: 1) Point cloud completion for the target object based on the partial observation; 2) Diverse sets of grasps generation given the complete point cloud; 3) Iterative grasp pose refinement for physically plausible grasp optimization. To train our model, we build a large-scale grasping dataset that contains about 300 common object models with 1.5M annotated grasps in simulation. Experiments in simulation show that our model can predict robust grasp poses with a wide variety and high success rate. Real robot platform experiments demonstrate that the model trained on our dataset performs well in the real world. Remarkably, our method achieves a grasp success rate of 70.7\% for novel objects in the real robot platform, which is a significant improvement over the baseline methods.
BibTeX
@article{wei2022dvgg,
  author = {Wei Wei and Daheng Li and Peng Wang and Yiming Li and Wanyi Li and Yongkang Luo and Jun Zhong},
  title = {{DVGG: Deep Variational Grasp Generation for Dextrous Manipulation}},
  journal = {IEEE RA-L},
  year = {2022},
  doi = {10.1109/lra.2022.3140424},
}
Figure