Abstract
The Embodied Dexterous Grasping System (EDGS) is a modular framework that enables robots to perform complex, human-like grasping in cluttered environments using only vague voice commands. By combining a Vision-Language Model (VLM) to enrich object descriptions with a human-inspired "thumb-finger coordination" grasping strategy, the system achieves a 96.1% success rate across diverse real-world household objects.
Introduction
While robotic dexterity and natural language processing have advanced significantly, connecting the two remains a "last mile" challenge. Current systems face two primary hurdles: semantic ambiguity in human voice commands and the computational complexity of planning stable grasps for irregular objects using multi-fingered hands. EDGS addresses these by mapping vague instructions to specific objects and employing a robust, human-centric grasping strategy.
System Architecture
The EDGS framework is divided into three primary modules: perception, policy generation, and proprioception.
Referring Expression Representation Enrichment (RERE)
Instead of relying on raw voice transcripts, EDGS uses a VLM to analyze the scene and the command simultaneously. It enriches object descriptions with attributes like color, material, and relative position (e.g., "the small bright yellow rubber duck on the left"). This context-aware approach significantly improves the accuracy of downstream segmentation models, avoiding "class confusion" between similar-looking objects.
Grasp Generation and Refinement
The system generates and refines grasp candidates through a multi-stage process inspired by human heuristics, prioritizing thumb-finger coordination.
Dexterous Grasp Candidates Generation (DGCG)
This module extracts a 2D "skeleton" of the target object using a thinning algorithm. It then samples potential grasp poses based on the object's principal axes, ensuring the thumb and fingers are positioned to effectively "wrap" the object. This skeleton-based approach allows the system to find the "pinching axis" more reliably than standard point-cloud methods.
Dexterous Grasp Refinement (DGR)
To ensure stability, candidates are filtered through force-closure validation and Grasp Wrench Space (GWS) analysis, which measures resistance to external disturbances. Finally, the system uses STOMP (Stochastic Trajectory Optimization for Motion Planning) to select the path that requires minimal effort, mimicking human movement efficiency.
Experimental Results
The researchers evaluated EDGS using GraspNet-1Billion, DexGraspNet 2.0, and a real-world custom set of 11 diverse household items.
- Segmentation Boost: Applying RERE to Grounded SAM nearly doubled its performance on the GraspNet-1Billion dataset, raising the IoU from 44.9 to 64.4.
- Superior Success Rates: In multi-object grasping tasks, EDGS achieved a 95.5% success rate, significantly outperforming existing SOTA methods like DexGraspNet 2.0 (90.7%) and ISAGrasp (54.8%).
- Real-World Robustness: Across fruits, household items, and vegetables, the system maintained an overall success rate of 96.1%, even when dealing with disorganized, "unseen" objects.
Limitations and Future Work
Despite its high success rates, the authors note that the system currently lacks fine-grained tactile sensing for delicate re-orientation and is physically limited to single-handed tasks. Future work aims to integrate haptic feedback and dual-arm coordination, as well as structural-to-functional mapping to understand object parts (e.g., grasping a handle vs. a blade).
Summary updated