Grasp as You Say: Language-guided Dexterous Grasp Generation

Yi-Lin Wei, Jian-Jian Jiang, Chengyi Xing, Xiantuo Tan, Xiao-Ming Wu, Hao Li, Mark R. Cutkosky, Wei-Shi Zheng

Introduction

Traditional robotic grasping research has historically prioritized stability—the mechanical assurance that an object will not be dropped. However, human grasping is fundamentally driven by intention; we adjust our hand configuration based on whether we intend to use a tool, pass it to another person, or simply move it. Bridging the gap between stability-centric grasping and intent-aligned manipulation remains a significant challenge in robotics.

This paper introduces DexGYS (Dexterous Grasp as You Say), a framework designed to generate complex, human-like grasps from natural language instructions. By decoupling the generation of intent from the enforcement of physical constraints, the researchers enable multi-fingered robotic hands to perform task-specific grasps that align with fine-grained textual commands.

The Challenge of Dexterous Manipulation

Generating grasps for multi-fingered (dexterous) hands is significantly more complex than for simple parallel-jaw grippers due to high degrees of freedom (DoF). Two primary hurdles exist:

  1. The Penetration Conflict: To ensure physical realism, models use "penetration loss" to prevent the robot's fingers from passing through solid objects. However, the authors identify a "Penetration Loss Paradox": applying these constraints too early in the training process often suppresses the diversity of poses, forcing the model toward a few "safe" but non-functional grasps.
  2. Language Integration: Most task-oriented grasping is limited to rigid categories (e.g., "pour"). Natural language offers more flexibility but requires a sophisticated mapping between semantic instructions and high-dimensional joint configurations.

The DexGYSGrasp Framework

The authors propose a two-stage "progressive" learning strategy called DexGYSGrasp to resolve the conflict between intent and physical feasibility.

1. Intention and Diversity Grasp Component (IDGC)

The first stage utilizes a Conditional Diffusion Model to generate a wide distribution of potential hand poses based on the input text (e.g., "place your forefinger on the trigger"). Crucially, penetration loss is ignored during this phase. This allows the model to explore the full semantic space of the instruction without being prematurely constrained by physical perfection, ensuring that the resulting "coarse" grasp is primarily aligned with the user's intent.

2. Quality Grasp Component (QGC)

The second stage is a regressive refinement process. It takes the coarse pose from the IDGC and applies rigorous physics-based losses, including penetration constraints, joint limits, and contact map alignment. This stage "cleans up" the pose, ensuring the final output is stable, physically possible, and maintains the intended contact points.

To facilitate training across different hardware, the researchers developed Hand-Object Interaction Retargeting (HOIR). This strategy maps human hand-object interactions onto various robotic platforms—such as the Shadow, Allegro, and Leap hands—while preserving the semantic integrity of the contact points.

DexGYSNet: A Large-Scale Dataset

To support this framework, the authors introduced DexGYSNet, a massive dataset comprising 50,000 high-quality dexterous grasp pairs across 1,800 household objects. To overcome the difficulty of manual labeling, they employed an LLM-assisted annotation system. By feeding raw contact data into GPT-3.5, they converted technical coordinates into natural, fine-grained human commands, such as "Grasp the handle securely with your thumb and index finger."

Experimental Results and Insights

The DexGYSGrasp framework was evaluated against state-of-the-art (SOTA) models like SceneDiffuser and DGTR. Key findings include:

  • Superior Alignment: The model demonstrated significantly tighter alignment with textual instructions, as measured by Chamfer Distance (CD) and Fréchet Inception Distance (FID).
  • Diversity of Solutions: Unlike baseline models that often collapsed into a single "safe" pose, DexGYSGrasp produced a diverse array of valid grasps for the same instruction.
  • Real-World Performance: In physical trials using an Allegro hand on a Flexiv Rizon 4 arm, the system achieved high success rates on complex objects, including a 9/10 success rate for a power drill and 8/10 for a game controller.

Conclusion

The DexGYS framework highlights that the decomposition of "what to do" (intent) from "how to do it safely" (quality) is essential for language-guided manipulation. While challenges remain in fine-motor coordination for highly specific tasks—such as operating a trigger sprayer—and the dependency on high-quality vision pipelines, this research marks a significant step toward robots that can understand and execute complex human instructions in everyday environments.

Summary updated

BibTeX
@inproceedings{wei2024grasp,
  author = {Yi-Lin Wei and Jian-Jian Jiang and Chengyi Xing and Xiantuo Tan and Xiao-Ming Wu and Hao Li and Mark R. Cutkosky and Wei-Shi Zheng},
  title = {{Grasp as You Say: Language-guided Dexterous Grasp Generation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=NR0oaKbqtF},
}
Figure