DDGC: Generative Deep Dexterous Grasping in Clutter

Jens Lundell, Francesco Verdoja, Ville KyrkiIEEE RA-L 2021 · DOI

TL;DR

This paper introduces DDGC, a novel deep learning approach for generating high-quality, collision-free multi-finger grasps in cluttered scenes from a single RGB-D image. DDGC significantly outperforms existing methods in both grasp quality and speed, making dexterous manipulation in complex environments more feasible.

What Problem Does This Paper Address?

Robotic grasping, especially with multi-finger hands, is challenging in cluttered environments. While parallel-jaw grippers have seen progress in clutter, multi-finger hands offer more dexterity but struggle with the increased complexity of reasoning about multiple objects and avoiding collisions. Existing methods for multi-finger grasping in clutter are either computationally expensive, requiring tens of seconds to minutes per grasp, or rely on extensive simulation setups that are brittle to tune. This paper aims to bridge this gap by enabling fast, high-quality multi-finger grasp generation in cluttered scenes from a single RGB-D input.

Methods

DDGC employs a generative deep network architecture that works in a coarse-to-fine manner. The core innovation lies in its ability to process cluttered scenes and generate dexterous grasps. The network consists of five sub-modules:

  • Scene Completion: Reconstructs a full 3D model of the scene by segmenting objects and completing their shapes. This is crucial for understanding the environment beyond the visible parts of objects.
  • Image Encoding: Processes the RGB image to create a scene geometry encoding. Notably, it uses an unmasked RGB image with an additional channel for the target object mask, allowing it to consider obstacles.
  • Grasp Generation: Takes the scene encoding and an input grasp orientation to produce a coarse gripper pose and hand joint configuration.
  • Discriminator: A Wasserstein discriminator is used to ensure the generated grasps are close to the training distribution.
  • Finger Refinement: A novel, fully differentiable layer that refines the coarse gripper configuration by rotating fingers until they touch the object, ensuring contact without requiring precise pre-grasp configurations.

The method is trained end-to-end using a combination of loss functions, including contact loss, collision loss, orientation loss, and plane loss, to encourage robust and collision-free grasps.

Datasets

The authors trained DDGC on purely simulated data. They generated datasets of cluttered scenes with 1 to 4 objects, using objects from the YCB object set and the EGAD! training set. Three validation sets were created: one from EGAD! validation objects, one from the KIT object model database, and one combining both. For real-world experiments, they used seven scenes with four objects each, manually placed to create occlusions.

Key Insights and Ideas

  • Scene Completion is Key for Clutter: By reconstructing the full 3D scene, DDGC can effectively reason about obstacles, a critical step for collision-free grasping in clutter.
  • Targeted Grasping via Masks: Using object masks allows DDGC to be target-driven, enabling users to specify which object to grasp, a significant advantage over target-agnostic methods.
  • Coarse-to-Fine with Finger Refinement: The two-stage approach (coarse generation followed by fine refinement) combined with a differentiable finger refinement layer is effective for generating precise, contact-rich grasps.
  • Simulation-to-Real Transfer: DDGC demonstrates strong generalization from simulated training data to real-world grasping without any fine-tuning, likely due to its object-centric grasp generation and translation invariance.
  • Speed vs. Quality Trade-off: DDGC achieves a remarkable balance, producing high-quality grasps significantly faster than previous methods, opening doors for practical applications.

Key Results

  • Simulation: DDGC generated significantly higher quality grasps (measured by e-quality) than both GraspIt! and Multi-FinGAN. It also showed much lower interpenetration.
  • Speed: DDGC was 4-5 times faster than GraspIt!, generating 100 grasps in under 10 seconds, compared to GraspIt!'s ~40 seconds.
  • Real-World Performance: On physical hardware, DDGC achieved a higher average clearance rate (71.43%) and grasp success rate (40.00%) compared to Multi-FinGAN and GraspIt!.
  • Clutter Robustness: DDGC's performance remained consistent with increasing clutter, unlike GraspIt!, which degraded significantly.

Limitations and Future Work

  • Sampling Speed: While faster than previous methods, sampling 100 grasps still takes several seconds, primarily due to the expensive collision checking. The authors suggest using a separate network for collision detection to improve speed.
  • Grasp Quality vs. Success: High simulated grasp quality doesn't always translate to successful real-world grasps. They propose exploring critique networks to score multi-finger grasps, a problem still open for dexterous hands.
  • Catastrophic Failures: DDGC can occasionally produce poor grasps, which might be addressed by priming the network to produce more "power type" grasps or by improving extrinsic calibration and friction modeling.

Summary updated

BibTeX
@article{lundell2021ddgc,
  author = {Jens Lundell and Francesco Verdoja and Ville Kyrki},
  title = {{DDGC: Generative Deep Dexterous Grasping in Clutter}},
  journal = {IEEE RA-L},
  year = {2021},
  doi = {10.1109/lra.2021.3096239},
}
Figure