Grasp What You Want: Embodied Dexterous Grasping System Driven by Your Voice

Junliang Li, Kai Ye, Haolan Kang, Mingxuan Liang, Yuhang Wu, Zhenhua Liu, Huiping Zhuang, Rui Huang, Yongquan ChenarXiv 2024 · DOI · arXiv

Abstract

The Embodied Dexterous Grasping System (EDGS) is a modular framework that enables robots to perform complex, human-like grasping in cluttered environments using only vague voice commands. By combining a Vision-Language Model (VLM) to enrich object descriptions with a human-inspired "thumb-finger coordination" grasping strategy, the system achieves a 96.1% success rate across diverse real-world household objects.

Introduction

While robotic dexterity and natural language processing have advanced significantly, connecting the two remains a "last mile" challenge. Current systems face two primary hurdles: semantic ambiguity in human voice commands and the computational complexity of planning stable grasps for irregular objects using multi-fingered hands. EDGS addresses these by mapping vague instructions to specific objects and employing a robust, human-centric grasping strategy.

System Architecture

The EDGS framework is divided into three primary modules: perception, policy generation, and proprioception.

Overview of the EDGS framework, illustrating the flow from voice command perception to policy generation and execution.

Referring Expression Representation Enrichment (RERE)

Instead of relying on raw voice transcripts, EDGS uses a VLM to analyze the scene and the command simultaneously. It enriches object descriptions with attributes like color, material, and relative position (e.g., "the small bright yellow rubber duck on the left"). This context-aware approach significantly improves the accuracy of downstream segmentation models, avoiding "class confusion" between similar-looking objects.

Grasp Generation and Refinement

The system generates and refines grasp candidates through a multi-stage process inspired by human heuristics, prioritizing thumb-finger coordination.

Detailed pipeline for feature extraction, grasp affordance assessment, and action refinement within the EDGS framework.

Dexterous Grasp Candidates Generation (DGCG)

This module extracts a 2D "skeleton" of the target object using a thinning algorithm. It then samples potential grasp poses based on the object's principal axes, ensuring the thumb and fingers are positioned to effectively "wrap" the object. This skeleton-based approach allows the system to find the "pinching axis" more reliably than standard point-cloud methods.

Dexterous Grasp Refinement (DGR)

To ensure stability, candidates are filtered through force-closure validation and Grasp Wrench Space (GWS) analysis, which measures resistance to external disturbances. Finally, the system uses STOMP (Stochastic Trajectory Optimization for Motion Planning) to select the path that requires minimal effort, mimicking human movement efficiency.

Experimental Results

The researchers evaluated EDGS using GraspNet-1Billion, DexGraspNet 2.0, and a real-world custom set of 11 diverse household items.

  • Segmentation Boost: Applying RERE to Grounded SAM nearly doubled its performance on the GraspNet-1Billion dataset, raising the IoU from 44.9 to 64.4.
  • Superior Success Rates: In multi-object grasping tasks, EDGS achieved a 95.5% success rate, significantly outperforming existing SOTA methods like DexGraspNet 2.0 (90.7%) and ISAGrasp (54.8%).
  • Real-World Robustness: Across fruits, household items, and vegetables, the system maintained an overall success rate of 96.1%, even when dealing with disorganized, "unseen" objects.

Limitations and Future Work

Despite its high success rates, the authors note that the system currently lacks fine-grained tactile sensing for delicate re-orientation and is physically limited to single-handed tasks. Future work aims to integrate haptic feedback and dual-arm coordination, as well as structural-to-functional mapping to understand object parts (e.g., grasping a handle vs. a blade).

Summary updated

BibTeX
@misc{li2024grasp,
  author = {Junliang Li and Kai Ye and Haolan Kang and Mingxuan Liang and Yuhang Wu and Zhenhua Liu and Huiping Zhuang and Rui Huang and Yongquan Chen},
  title = {{Grasp What You Want: Embodied Dexterous Grasping System Driven by Your Voice}},
  year = {2024},
  eprint = {2412.10694},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.10694},
}
Figure