CoPa: General Robotic Manipulation through Spatial Constraints of Parts with Foundation Models

Haoxu Huang, Fanqi Lin, Yingdong Hu, Shengjie Wang, Yang Gao

Abstract

Recent advancements in foundation models have significantly improved high-level robotic task planning. However, bridging the gap between high-level reasoning and low-level physical execution remains a challenge. We present CoPa (Spatial Constraints of Parts), a framework that leverages Vision-Language Models (VLMs) to decompose complex manipulation tasks into part-level geometric constraints. By identifying task-relevant object parts and defining their spatial relationships, CoPa enables zero-shot, open-set robotic manipulation without task-specific training.

Introduction

While large-scale foundation models like GPT-4V have demonstrated remarkable "common sense" reasoning capabilities, applying this knowledge to precise robotic control is difficult. High-level planners can decide what needs to be done (e.g., "hammer the nail"), but low-level policies often struggle with the how—the precise 6-DoF movements required to interact with objects. Traditional methods rely on imitation learning or reinforcement learning, which are data-intensive and often fail to generalize to novel objects. CoPa addresses this by using VLMs to extract fine-grained physical understanding, focusing on the geometric constraints of specific object parts rather than treating objects as holistic entities.

The Problem: The Low-Level Control Bottleneck

The primary bottleneck in modern robotics is the lack of fine-grained physical understanding during execution. For instance, a robot tasked with using a hammer must understand that it should grasp the handle, not the head, and that the striking surface must align with the nail. Current low-level policies typically lack this semantic awareness. CoPa proposes that the missing link is the ability to translate semantic knowledge into spatial constraints that a mathematical solver can optimize.

Methodology

CoPa operates through a two-phase process: Task-Oriented Grasping and Task-Aware Motion Planning.

1. Task-Oriented Grasping

To ensure a functional grasp, CoPa employs a coarse-to-fine grounding strategy. Using the Set-of-Mark (SoM) prompting technique, the system first identifies the target object and then identifies the specific part suitable for the task (e.g., the handle of a kettle). Once the part is masked, the system utilizes GraspNet to generate thousands of potential 6-DoF grasp poses. These poses are filtered using the VLM-identified mask to select the most functional grasp for the intended action.

2. Task-Aware Motion Planning

Instead of requiring the VLM to output precise numerical coordinates—a task for which they are poorly suited—CoPa asks the model to describe the task through spatial constraints. For example, in a pouring task, the VLM might specify that the spout of a kettle must be positioned above a cup and then rotated.

These textual constraints are translated into geometric primitives, such as vectors for slender parts or surfaces for flat ones. These primitives are then converted into mathematical loss functions. A non-linear optimizer solves for the exact 6-DoF end-effector poses that satisfy these constraints, providing a bridge between semantic intent and physical motion.

Experimental Evaluation

CoPa was evaluated as a zero-shot framework on a real-world benchmark consisting of 10 diverse manipulation tasks, including hammering a nail, pouring water, inserting a flower into a vase, and sweeping. Each task involved 10 variations with different object types and arrangements to test the system's generalization capabilities.

Key Results

  • Success Rate: CoPa achieved an overall success rate of 63%, significantly outperforming the baseline method, VoxPoser, which achieved only 18%.
  • Rotational Precision: The framework excelled in tasks requiring complex 3D rotations, such as orienting a hammer or pouring, where previous methods often failed due to treating objects as single rigid points.
  • Ablation Study: When the foundation model was removed, the success rate dropped to 11%, confirming that the VLM’s semantic and physical knowledge is the core driver of performance.

Discussion and Limitations

CoPa demonstrates that spatial constraints serve as an effective interface between high-level reasoning and low-level control. By focusing on part-level awareness, the system achieves human-like intentionality in its movements with minimal prompting compared to previous methods.

However, limitations remain. Currently, CoPa models parts as simple geometric primitives (vectors or surfaces), which may not suffice for highly irregular 3D shapes. Additionally, because models like GPT-4V are trained on 2D images, the projection to 3D space can occasionally introduce spatial errors. Future work will explore the integration of native 3D foundation models and a broader library of geometric primitives to enhance robustness in complex environments.

Conclusion

CoPa provides a scalable approach to open-set robotic manipulation by leveraging the common-sense knowledge of VLMs to define spatial constraints. By shifting the focus from raw coordinate prediction to geometric reasoning, CoPa moves robotics closer to achieving general-purpose manipulation capabilities in unstructured, real-world settings.

Summary updated

Abstract

Foundation models pre-trained on web-scale data are shown to encapsulate extensive world knowledge beneficial for robotic manipulation in the form of task planning. However, the actual physical implementation of these plans often relies on task-specific learning methods, which require significant data collection and struggle with generalizability. In this work, we introduce Robotic Manipulation through Spatial Constraints of Parts (CoPa), a novel framework that leverages the common sense knowledge embedded within foundation models to generate a sequence of 6-DoF end-effector poses for open-world robotic manipulation. Specifically, we decompose the manipulation process into two phases: task-oriented grasping and task-aware motion planning. In the task-oriented grasping phase, we employ foundation vision-language models (VLMs) to select the object’s grasping part through a novel coarse-to-fine grounding mechanism. During the task-aware motion planning phase, VLMs are utilized again to identify the spatial geometry constraints of task-relevant object parts, which are then used to derive post-grasp poses. We also demonstrate how COPA can be seamlessly integrated with existing robotic planning algorithms to accomplish complex, long-horizon tasks. Our comprehensive real-world experiments show that COPA possesses a fine-grained physical understanding of scenes, capable of handling open-set instructions and objects with minimal prompt engineering and without additional training.
BibTeX
@inproceedings{huang2024copa,
  author = {Haoxu Huang and Fanqi Lin and Yingdong Hu and Shengjie Wang and Yang Gao},
  title = {{CoPa: General Robotic Manipulation through Spatial Constraints of Parts with Foundation Models}},
  booktitle = {VLMNM @ ICRA},
  year = {2024},
  url = {https://openreview.net/forum?id=HOtu0qfBL1},
}
Figure