Abstract
Recent advancements in foundation models have significantly improved high-level robotic task planning. However, bridging the gap between high-level reasoning and low-level physical execution remains a challenge. We present CoPa (Spatial Constraints of Parts), a framework that leverages Vision-Language Models (VLMs) to decompose complex manipulation tasks into part-level geometric constraints. By identifying task-relevant object parts and defining their spatial relationships, CoPa enables zero-shot, open-set robotic manipulation without task-specific training.
Introduction
While large-scale foundation models like GPT-4V have demonstrated remarkable "common sense" reasoning capabilities, applying this knowledge to precise robotic control is difficult. High-level planners can decide what needs to be done (e.g., "hammer the nail"), but low-level policies often struggle with the how—the precise 6-DoF movements required to interact with objects. Traditional methods rely on imitation learning or reinforcement learning, which are data-intensive and often fail to generalize to novel objects. CoPa addresses this by using VLMs to extract fine-grained physical understanding, focusing on the geometric constraints of specific object parts rather than treating objects as holistic entities.
The Problem: The Low-Level Control Bottleneck
The primary bottleneck in modern robotics is the lack of fine-grained physical understanding during execution. For instance, a robot tasked with using a hammer must understand that it should grasp the handle, not the head, and that the striking surface must align with the nail. Current low-level policies typically lack this semantic awareness. CoPa proposes that the missing link is the ability to translate semantic knowledge into spatial constraints that a mathematical solver can optimize.
Methodology
CoPa operates through a two-phase process: Task-Oriented Grasping and Task-Aware Motion Planning.
1. Task-Oriented Grasping
To ensure a functional grasp, CoPa employs a coarse-to-fine grounding strategy. Using the Set-of-Mark (SoM) prompting technique, the system first identifies the target object and then identifies the specific part suitable for the task (e.g., the handle of a kettle). Once the part is masked, the system utilizes GraspNet to generate thousands of potential 6-DoF grasp poses. These poses are filtered using the VLM-identified mask to select the most functional grasp for the intended action.
2. Task-Aware Motion Planning
Instead of requiring the VLM to output precise numerical coordinates—a task for which they are poorly suited—CoPa asks the model to describe the task through spatial constraints. For example, in a pouring task, the VLM might specify that the spout of a kettle must be positioned above a cup and then rotated.
These textual constraints are translated into geometric primitives, such as vectors for slender parts or surfaces for flat ones. These primitives are then converted into mathematical loss functions. A non-linear optimizer solves for the exact 6-DoF end-effector poses that satisfy these constraints, providing a bridge between semantic intent and physical motion.
Experimental Evaluation
CoPa was evaluated as a zero-shot framework on a real-world benchmark consisting of 10 diverse manipulation tasks, including hammering a nail, pouring water, inserting a flower into a vase, and sweeping. Each task involved 10 variations with different object types and arrangements to test the system's generalization capabilities.
Key Results
- Success Rate: CoPa achieved an overall success rate of 63%, significantly outperforming the baseline method, VoxPoser, which achieved only 18%.
- Rotational Precision: The framework excelled in tasks requiring complex 3D rotations, such as orienting a hammer or pouring, where previous methods often failed due to treating objects as single rigid points.
- Ablation Study: When the foundation model was removed, the success rate dropped to 11%, confirming that the VLM’s semantic and physical knowledge is the core driver of performance.
Discussion and Limitations
CoPa demonstrates that spatial constraints serve as an effective interface between high-level reasoning and low-level control. By focusing on part-level awareness, the system achieves human-like intentionality in its movements with minimal prompting compared to previous methods.
However, limitations remain. Currently, CoPa models parts as simple geometric primitives (vectors or surfaces), which may not suffice for highly irregular 3D shapes. Additionally, because models like GPT-4V are trained on 2D images, the projection to 3D space can occasionally introduce spatial errors. Future work will explore the integration of native 3D foundation models and a broader library of geometric primitives to enhance robustness in complex environments.
Conclusion
CoPa provides a scalable approach to open-set robotic manipulation by leveraging the common-sense knowledge of VLMs to define spatial constraints. By shifting the focus from raw coordinate prediction to geometric reasoning, CoPa moves robotics closer to achieving general-purpose manipulation capabilities in unstructured, real-world settings.
Summary updated