ABM: Attention before Manipulation

Fan Zhuo, Ying He, Fei Yu, Pengteng Li, Zheyi Zhao, Xilong Sun

Abstract

Vision-language models (VLMs) show promising generalization and zero-shot capabilities, offering a potential solution to the impracticality and cost of enabling robots to comprehend diverse human instructions and scene semantics in the real world. Existing approaches most directly integrate the semantic representations from pre-trained VLMs with policy learning. However, these methods are limited to the labeled data learned, resulting in poor generalization ability to unseen instructions and objects. To address the above limitation, we propose a simple method called "Attention before Manipulation" (ABM), which fully leverages the object knowledge encoded in CLIP to extract information about the target object in the image. It constructs an Object Mask Field, serving as a better representation of the target object for the model to separate visual grounding from action prediction and acquire specific manipulation skills effectively. We train ABM for 8 RLBench tasks and 2 real-world tasks via behavior cloning. Extensive experiments show that our method significantly outperforms the baselines in the zero-shot and compositional generalization experiment settings.

Introduction: The "Binding" Problem in Robot Learning

Current vision-language models (VLMs) for robotics often suffer from a "binding" problem. When trained end-to-end, models tightly couple specific instructions, visual perceptions, and robot actions. For instance, a robot might learn to "pick up a red block," but fail when asked to pick up a "purple block" or a "tomato" because it hasn't truly decoupled the concept of the object from the skill of picking.

ABM demonstrates generalization to unseen objects (Zero-Shot) and novel skill-object pairings (Compositional) compared to training examples.

Existing methods that directly feed VLM features into a policy often only achieve "feature-level" generalization—they work if the new data is semantically similar to training data but fail in truly "zero-shot" scenarios where objects or instructions are completely novel.

Methodology: Decoupling Perception and Action

ABM (Attention before Manipulation) is a novel framework that improves how robots follow human instructions by decoupling "finding the object" from "performing the task." The authors propose a two-step approach:

  1. Instruction Decomposition: Instead of treating the whole command as one goal, ABM splits it into a target object description (e.g., "tomato") and a robot skill (e.g., "put X in the plate").
  2. Object Mask Field (OMF):
    • The system uses a frozen CLIP model to extract dense features from the robot's camera views.
  3. These features are projected into 3D space to create a "CLIP Feature Cloud."
    • The system queries this cloud with the target object description to generate a 3D heatmap of where the object likely is.
  4. This heatmap is converted into a binary Object Mask Field, which serves as a spatial prior.
  5. Policy Architecture: ABM builds on the Robotic View Transformer (RVT). It re-renders the 3D scene and the OMF into 2D virtual images. Crucially, it simplifies the mask into a "center-pixel mask" to reduce noise from imperfect segmentation. The policy then focuses solely on manipulating the area indicated by the mask, using the "robot skill" as its linguistic goal.

Experimental Evaluation

The model was evaluated across several benchmarks:

  • RLBench: 8 tasks modified to include colors, quantities, and categories never seen during training.
  • Compositional Tasks: Two tasks combining skills and objects from different training sets (e.g., "place" skill with a "block" object).
  • Real-World Data: A custom dataset of 40 demonstrations using a UR3 robot arm for tasks like "place the toy in the cabinet."

Key Results

  • Zero-Shot Generalization: In simulation, ABM achieved a 57.5% success rate on unseen objects, significantly outperforming the standard RVT baseline (24%) and a baseline integrating CLIP features directly (34.5%).
  • Compositional Generalization: ABM reached a 53.2% success rate when combining skills and objects from different tasks, compared to 21.6% for the baseline.
  • Real-World Performance: Despite minimal training (40 demos), the model achieved a 65% success rate on unseen test cases.

Discussion and Limitations

By translating a complex instruction into a spatial mask before manipulation starts, the policy avoids learning visual grounding from scratch and focuses on the geometry of the action. However, constructing the OMF at every timestep is computationally expensive, resulting in a low inference rate (~0.68 FPS). Future work may involve keypoint tracking to improve efficiency. Additionally, the current model handles only one target object per task and relies heavily on the accuracy of CLIP's internal knowledge for grounding.

Summary updated

Abstract

Vision-language models (VLMs) show promising generalization and zero-shot capabilities, offering a potential solution to the impracticality and cost of enabling robots to comprehend diverse human instructions and scene semantics in the real world. Existing approaches most directly integrate the semantic representations from pre-trained VLMs with policy learning. However, these methods are limited to the labeled data learned, resulting in poor generalization ability to unseen instructions and objects. To address the above limitation, we propose a simple method called "Attention before Manipulation" (ABM), which fully leverages the object knowledge encoded in CLIP to extract information about the target object in the image. It constructs an Object Mask Field, serving as a better representation of the target object for the model to separate visual grounding from action prediction and acquire specific manipulation skills effectively. We train ABM for 8 RLBench tasks and 2 real-world tasks via behavior cloning. Extensive experiments show that our method significantly outperforms the baselines in the zero-shot and compositional generalization experiment settings.
BibTeX
@inproceedings{zhuo2024abm,
  author = {Fan Zhuo and Ying He and Fei Yu and Pengteng Li and Zheyi Zhao and Xilong Sun},
  title = {{ABM: Attention before Manipulation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=I9i1nIhC4W},
}

Loading related content...

Figure