TL;DR
UniDiffGrasp is a unified framework that bridges the gap between high-level semantic reasoning and low-level robotic precision. By combining Vision-Language Models (VLMs) for task understanding with "Part-Guided Diffusion" for grasp generation, it enables robots to perform precise, part-specific grasping (e.g., "pick up the pot by its handles") in open-vocabulary environments using both single and dual-arm configurations.
What Problem Does This Paper Address?
While modern robots are getting better at seeing objects, they still struggle with task-oriented grasping—the ability to grasp an object by a specific functional part (an "affordance") based on a natural language command. For example, a robot shouldn't just grab a screwdriver anywhere; it needs to grab the handle to use it.
Current solutions face a "semantic-geometric gap." Vision-Language Models (VLMs) are excellent at reasoning ("I need to find the handle") but poor at generating the precise 6-DoF (degrees of freedom) coordinates needed for a stable grasp. Conversely, specialized grasp generators are great at finding stable finger placements but often lack the semantic context to know which part of the object is relevant to the user's goal. This challenge is even greater for dual-arm robots, which must coordinate two hands to lift large or heavy objects by specific regions.
Methods
UniDiffGrasp operates through a multi-stage pipeline that integrates reasoning, grounding, and generative physics:
- VLM Reasoning: The system uses GPT-4o to interpret a user's natural language instruction and an RGB image. It identifies the target object, the specific functional part (e.g., "handle"), and the operation mode (single or dual-arm).
- Hierarchical Grounding: To turn text into geometry, the framework uses GroundedSAM to isolate the object and VLPart to segment the specific functional part. This results in a "target region" point cloud.
- Part-Guided Diffusion: This is the core technical innovation. Instead of using a standard grasp detector, they employ Constrained Grasp Diffusion Fields (CGDF). This generative model treats grasp synthesis as a denoising process. Crucially, it uses the VLM-identified part as a geometric constraint to "steer" the diffusion process toward the desired region during inference—without requiring any task-specific retraining.
- Dual-Arm Coordination: For large objects, the system defines two target regions via "Semantic Splitting" (finding two distinct handles) or "Geometric Splitting" (dividing the object into halves). It generates candidate grasps for each arm independently and then uses a selection module to find the best pair based on Force Closure (physical stability), collision checking, and spatial balance.
Datasets
The generative backbone (CGDF) is trained on unconstrained grasp data, but the paper's primary contribution is evaluated through extensive real-world experiments rather than just simulation.
- Single-Arm Evaluation: 7 diverse household objects (Bottle, Knife, Screwdriver, Pan, Mug, Pliers, Kettle).
- Dual-Arm Evaluation: 4 large-scale categories (Basin, Keyboard, Laptop, Pot).
- Baselines: Compared against GroundedSAM + GraspNet and ThinkGrasp (a state-of-the-art part-grasping baseline).
Key Insights and Ideas
- Zero-Shot Constraints: The most powerful idea here is that you don't need a dataset of "how to grasp a kettle handle" to grasp a kettle handle. By using diffusion models, the VLM's semantic output can be treated as a mathematical constraint that guides a general-purpose grasp model toward a specific area.
- Hierarchical Perception: By segmenting the object first and then the part, the model avoids "distractor" parts on other nearby objects, a common failure point in cluttered robotic environments.
- Decoupling Reasoning from Execution: The framework allows the "brain" (VLM) to handle the what and where, while the "motor system" (Diffusion) handles the how, ensuring that the final grasp is both semantically correct and physically stable.
Key Results
The researchers deployed UniDiffGrasp on a physical Baxter robot and achieved significant performance gains:
- Single-Arm Success: Achieved a Grasp Success Rate (GSR) of 0.876, significantly outperforming ThinkGrasp (0.705).
- Geometric Precision: The Collision-Free Rate (CFR) reached 0.900, indicating that the diffusion process is highly effective at respecting the complex geometry of functional parts.
- Dual-Arm Performance: In complex bimanual tasks, it achieved a GSR of 0.767, nearly doubling the performance of the baseline (0.475), which lacked semantic guidance for region definition.
Limitations and Future Work
- Perception Dependency: The system is heavily reliant on the upstream VLM and segmentation models. If VLPart fails to identify a handle correctly, the grasp generation will likely fail.
- Static Scenes: The current framework is designed for static objects. Adapting this to dynamic environments (e.g., grasping a moving object) remains a challenge.
- Bimanual Complexity: While the system coordinates two arms for a single lift, it does not yet address complex bimanual manipulation tasks, such as unscrewing a jar lid or folding cloth, which require more intricate temporal coordination.
Summary updated