UniDiffGrasp: A Unified Framework Integrating VLM Reasoning and VLM-Guided Part Diffusion for Open-Vocabulary Constrained Grasping with Dual Arms

Xueyang Guo, Hongwei Hu, Chengye Song, Jiale Chen, Zilin Zhao, Yu Fu, Bowen Guan, Zhenze LiuarXiv 2025 · DOI · arXiv

TL;DR

UniDiffGrasp is a unified framework that bridges the gap between high-level semantic reasoning and low-level robotic precision. By combining Vision-Language Models (VLMs) for task understanding with "Part-Guided Diffusion" for grasp generation, it enables robots to perform precise, part-specific grasping (e.g., "pick up the pot by its handles") in open-vocabulary environments using both single and dual-arm configurations.

What Problem Does This Paper Address?

While modern robots are getting better at seeing objects, they still struggle with task-oriented grasping—the ability to grasp an object by a specific functional part (an "affordance") based on a natural language command. For example, a robot shouldn't just grab a screwdriver anywhere; it needs to grab the handle to use it.

Current solutions face a "semantic-geometric gap." Vision-Language Models (VLMs) are excellent at reasoning ("I need to find the handle") but poor at generating the precise 6-DoF (degrees of freedom) coordinates needed for a stable grasp. Conversely, specialized grasp generators are great at finding stable finger placements but often lack the semantic context to know which part of the object is relevant to the user's goal. This challenge is even greater for dual-arm robots, which must coordinate two hands to lift large or heavy objects by specific regions.

Methods

UniDiffGrasp operates through a multi-stage pipeline that integrates reasoning, grounding, and generative physics:

  1. VLM Reasoning: The system uses GPT-4o to interpret a user's natural language instruction and an RGB image. It identifies the target object, the specific functional part (e.g., "handle"), and the operation mode (single or dual-arm).
  2. Hierarchical Grounding: To turn text into geometry, the framework uses GroundedSAM to isolate the object and VLPart to segment the specific functional part. This results in a "target region" point cloud.
  3. Part-Guided Diffusion: This is the core technical innovation. Instead of using a standard grasp detector, they employ Constrained Grasp Diffusion Fields (CGDF). This generative model treats grasp synthesis as a denoising process. Crucially, it uses the VLM-identified part as a geometric constraint to "steer" the diffusion process toward the desired region during inference—without requiring any task-specific retraining.
  4. Dual-Arm Coordination: For large objects, the system defines two target regions via "Semantic Splitting" (finding two distinct handles) or "Geometric Splitting" (dividing the object into halves). It generates candidate grasps for each arm independently and then uses a selection module to find the best pair based on Force Closure (physical stability), collision checking, and spatial balance.

Datasets

The generative backbone (CGDF) is trained on unconstrained grasp data, but the paper's primary contribution is evaluated through extensive real-world experiments rather than just simulation.

  • Single-Arm Evaluation: 7 diverse household objects (Bottle, Knife, Screwdriver, Pan, Mug, Pliers, Kettle).
  • Dual-Arm Evaluation: 4 large-scale categories (Basin, Keyboard, Laptop, Pot).
  • Baselines: Compared against GroundedSAM + GraspNet and ThinkGrasp (a state-of-the-art part-grasping baseline).

Key Insights and Ideas

  • Zero-Shot Constraints: The most powerful idea here is that you don't need a dataset of "how to grasp a kettle handle" to grasp a kettle handle. By using diffusion models, the VLM's semantic output can be treated as a mathematical constraint that guides a general-purpose grasp model toward a specific area.
  • Hierarchical Perception: By segmenting the object first and then the part, the model avoids "distractor" parts on other nearby objects, a common failure point in cluttered robotic environments.
  • Decoupling Reasoning from Execution: The framework allows the "brain" (VLM) to handle the what and where, while the "motor system" (Diffusion) handles the how, ensuring that the final grasp is both semantically correct and physically stable.

Key Results

The researchers deployed UniDiffGrasp on a physical Baxter robot and achieved significant performance gains:

  • Single-Arm Success: Achieved a Grasp Success Rate (GSR) of 0.876, significantly outperforming ThinkGrasp (0.705).
  • Geometric Precision: The Collision-Free Rate (CFR) reached 0.900, indicating that the diffusion process is highly effective at respecting the complex geometry of functional parts.
  • Dual-Arm Performance: In complex bimanual tasks, it achieved a GSR of 0.767, nearly doubling the performance of the baseline (0.475), which lacked semantic guidance for region definition.

Limitations and Future Work

  • Perception Dependency: The system is heavily reliant on the upstream VLM and segmentation models. If VLPart fails to identify a handle correctly, the grasp generation will likely fail.
  • Static Scenes: The current framework is designed for static objects. Adapting this to dynamic environments (e.g., grasping a moving object) remains a challenge.
  • Bimanual Complexity: While the system coordinates two arms for a single lift, it does not yet address complex bimanual manipulation tasks, such as unscrewing a jar lid or folding cloth, which require more intricate temporal coordination.

Summary updated

Abstract

Open-vocabulary, task-oriented grasping of specific functional parts, particularly with dual arms, remains a key challenge, as current Vision-Language Models (VLMs), while enhancing task understanding, often struggle with precise grasp generation within defined constraints and effective dual-arm coordination. We innovatively propose UniDiffGrasp, a unified framework integrating VLM reasoning with guided part diffusion to address these limitations. UniDiffGrasp leverages a VLM to interpret user input and identify semantic targets (object, part(s), mode), which are then grounded via open-vocabulary segmentation. Critically, the identified parts directly provide geometric constraints for a Constrained Grasp Diffusion Field (CGDF) using its Part-Guided Diffusion, enabling efficient, high-quality 6-DoF grasps without retraining. For dual-arm tasks, UniDiffGrasp defines distinct target regions, applies part-guided diffusion per arm, and selects stable cooperative grasps. Through extensive real-world deployment, UniDiffGrasp achieves grasp success rates of 0.876 in single-arm and 0.767 in dual-arm scenarios, significantly surpassing existing state-of-the-art methods, demonstrating its capability to enable precise and coordinated open-vocabulary grasping in complex real-world scenarios.
BibTeX
@misc{guo2025unidiffgrasp,
  author = {Xueyang Guo and Hongwei Hu and Chengye Song and Jiale Chen and Zilin Zhao and Yu Fu and Bowen Guan and Zhenze Liu},
  title = {{UniDiffGrasp: A Unified Framework Integrating VLM Reasoning and VLM-Guided Part Diffusion for Open-Vocabulary Constrained Grasping with Dual Arms}},
  year = {2025},
  eprint = {2505.06832},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.06832},
  howpublished = {arXiv},
}
Figure