Abstract
End-to-end robot learning models often struggle with generalization, frequently overfitting to specific object-instruction pairings seen during training. We present Attention before Manipulation (ABM), a framework designed to decouple visual grounding from action prediction. By utilizing frozen Vision-Language Models (CLIP) to generate a 3D Object Mask Field (OMF), ABM enables robots to apply learned physical skills to entirely novel semantic categories. Our evaluation on the RLBench benchmark and real-world tasks demonstrates that ABM significantly improves zero-shot and compositional generalization, outperforming existing feature-integration methods by over 20% in success rate.
Introduction
In the field of robot manipulation, end-to-end models are highly sought after for their simplicity. However, these models often fail to generalize because they conflate the task of identifying an object with the physical skill required to manipulate it. For example, a robot trained to "pick up the red block" may fail to "pick up the purple block" if it hasn't learned to separate the concept of color from the act of picking.
Existing approaches that incorporate Vision-Language Models (VLMs) like CLIP typically do so at the feature level, which often results in "feature-level generalization"—the robot only succeeds if the test scene closely resembles the training data. The ABM framework addresses this limitation by explicitly decoupling visual grounding (finding the object) from action prediction (moving the robotic arm).
The ABM Framework
The ABM framework operates through a structured three-stage pipeline:
- Instruction Decomposition: Human commands (e.g., "Put the banana in the plate") are parsed into two components: the Target Object ("banana") and the Robot Skill ("Put X in the plate"). This can be achieved using simple regular expressions or Large Language Models like ChatGPT.
- Object Mask Field (OMF) Construction: The system extracts patch-level features from a frozen CLIP image encoder and projects them into a 3D point cloud of the environment. By querying this field with the Target Object text, the system generates a 3D heatmap (the OMF) that identifies the object's spatial coordinates based on CLIP’s extensive pre-trained knowledge.
- Policy Learning via Multi-view Transformer: The 3D OMF is re-rendered into 2D "center-pixel masks" from five virtual viewpoints. These masks are fused with standard RGB-D images and processed by a transformer-based policy. Because the policy is conditioned on a spatial mask rather than raw semantic features, it focuses solely on learning the physical skill relative to the mask's position.
Experimental Evaluation
The researchers evaluated ABM's performance using the RLBench manipulation benchmark and real-world experiments on a UR3 robot arm. The study focused on two primary challenges to generalization:
- Zero-Shot Validation: Testing the robot on tasks involving colors, object quantities, and categories never encountered during the training phase.
- Compositional Tasks: Evaluating the robot's ability to combine a skill learned from one task (e.g., "put in plate") with an object learned from another (e.g., "purple block").
Results and Key Insights
ABM demonstrated a marked superiority over baseline methods in both simulated and real-world environments:
- Zero-Shot Performance: In simulation, ABM outperformed the baseline Robotic View Transformer (RVT) by 33.5% and a feature-integration method (IFD) by 23% on unseen objects.
- Compositional Generalization: ABM achieved a 53.2% success rate on compositional tasks, whereas the standard RVT struggled at 21.6%.
- Real-World Robustness: On a physical UR3 robot, ABM maintained a 65% success rate on unseen test tasks, confirming the effectiveness of the "Attention before Manipulation" strategy.
Key Insights
- Decoupling is Essential: By converting the "where" into a spatial mask before the policy processes the scene, the robot avoids the need to re-learn object identification for every new task. It simply applies its physical skills to the coordinates provided by the mask.
- The Power of the Center-Pixel Mask: The authors discovered that a single pixel at the center of the detected object provided a more stable and effective target for the policy than a full, potentially noisy segmentation mask.
- Leveraging Frozen Knowledge: Keeping the CLIP backbone frozen prevents the model from overfitting to the relatively small robotic datasets, preserving its broad semantic understanding.
Conclusion and Future Work
ABM significantly improves robot generalization by leveraging large-scale vision-language models to separate perception from action. While the current system's inference speed (0.68 FPS) is a bottleneck for high-speed reactive tasks, the results clearly indicate that decoupling visual grounding is a promising direction for zero-shot robot learning. Future research will focus on optimizing inference through keypoint tracking and extending the framework to handle complex, multi-step instructions.
Summary updated