TL;DR
This paper introduces ManipLVM-R1, a novel reinforcement learning framework for robotic manipulation that leverages verifiable rewards instead of expensive human-annotated data. By designing specialized reward functions for affordance perception and trajectory prediction, ManipLVM-R1 achieves significant performance gains and improved generalization, even with less training data and in out-of-domain scenarios.
What Problem Does This Paper Address?
Large Vision-Language Models (LVLMs) have shown great promise in understanding visual scenes and following language instructions. However, applying them to robotic manipulation, which requires precise physical interaction and reasoning, has been hindered by a reliance on costly, human-annotated datasets. This dependency limits the models' ability to generalize to new situations and environments (out-of-domain or OOD scenarios), making them less adaptable for real-world applications. Existing methods often struggle with fine-grained control and understanding physical dynamics, leading to shallow pattern matching rather than robust reasoning.
Methods
ManipLVM-R1 tackles these challenges by adopting a Reinforcement Learning with Verifiable Rewards (RLVR) approach. Instead of learning from pre-collected demonstrations, the model learns by optimizing directly for task-aligned outcomes through carefully designed reward functions. The framework decomposes the manipulation task into two key subtasks:
- Affordance Perception Reward: This reward function focuses on accurately identifying where an object can be interacted with. It combines a format reward (ensuring the output is structured correctly) with an affordance reward that uses the Intersection-over-Union (IoU) metric to measure the overlap between the predicted and ground-truth bounding boxes of interaction regions. This encourages spatially precise and semantically grounded predictions.
- Trajectory Match Reward: This reward function aims to ensure the physical plausibility and goal-alignment of predicted action paths. It incorporates three components:
- Format Reward: Ensures the trajectory output adheres to a specific structure.
- Path Similarity Reward: Uses a weighted aggregation of metrics like Discrete Fréchet Distance (DFD), Hausdorff Distance (HD), and Root Mean Square Error (RMSE) to evaluate the overall shape, continuity, and alignment of the predicted trajectory with the ground truth. This goes beyond just endpoint accuracy.
- Endpoint Distance Reward: Measures the Euclidean distance between the predicted and ground-truth final points of the trajectory.
The model is trained using a Group Relative Policy Optimization (GRPO) inspired approach, where multiple responses are sampled and their advantages are calculated based on the assigned rewards. A KL divergence constraint is used to maintain stability during training.
Datasets
- In-Domain Dataset: The models were trained and evaluated on the ShareRobot dataset. This dataset is designed for affordance perception and trajectory prediction, containing multi-dimensional annotations like low-level planning instructions, object affordances, and end-effector trajectories. It comprises over 1M planning QA pairs, 6.5k affordance-annotated images, and 6.8k trajectory-annotated images across diverse scenarios. ManipLVM-R1 used only 50% of this dataset for training, while baselines used 100%.
- Out-of-Domain (OOD) Datasets:
- Affordance Perception: Subsets of the UMD Part Affordance dataset were used, focusing on categories like 'grasp', 'cut', 'pound', and 'scoop'.
- Trajectory Prediction: A randomly selected subset of 500 samples from the validation data of LLARVA's pre-training dataset, VAIT, was used. This dataset was manually verified and corrected for noise and inaccuracies.
Key Insights and Ideas
- RLVR for Robotics is Viable: The paper demonstrates that Reinforcement Learning with Verifiable Rewards (RLVR) can be effectively applied to robotic manipulation, overcoming the limitations of traditional supervised learning methods that rely on expensive human annotations.
- Structured Rewards Drive Better Reasoning: The design of specific, structured reward functions for affordance perception and trajectory prediction is crucial. These rewards impose spatial-logical constraints, pushing the model towards deeper physical reasoning rather than superficial pattern matching.
- Sample Efficiency Matters: By using only 50% of the training data, ManipLVM-R1 shows that RLVR can lead to more efficient learning, achieving comparable or better performance than models trained with full supervision.
- Generalization is Key: The framework's strong performance on OOD datasets highlights its ability to generalize beyond its training distribution, a critical factor for real-world robotic deployment.
- Multi-Metric Rewards for Trajectories: Combining multiple metrics (DFD, HD, RMSE) for trajectory evaluation provides a more comprehensive and stable learning signal compared to single metrics or simpler approaches like Dynamic Time Warping (DTW).
Key Results
- In-Domain Performance: ManipLVM-R1 achieved an IoU score of 31.0 on affordance perception, significantly outperforming baselines like RoboBrain-7B (11.79), despite using half the training data. On trajectory prediction, it achieved an average distance error of 110.87, competitive with fully supervised models.
- Out-of-Domain Performance: ManipLVM-R1 demonstrated robust generalization, achieving the highest Grasp-IoU (34.65) on the UMD affordance dataset and the lowest average trajectory error (131.99) on the VAIT benchmark, outperforming all evaluated open-source and supervised fine-tuned models.
- Qualitative Analysis: Visual examples showed the model successfully identifying affordances and generating plausible trajectories, even demonstrating emergent reasoning in some cases. Failure cases highlighted areas where commonsense knowledge or self-correction mechanisms could be beneficial.
Limitations and Future Work
- Focus on 2D: The current framework primarily focuses on affordance perception and 2D trajectory prediction. Real-world manipulation often involves more complex skills like fine-grained force control, deformable object manipulation, and dynamic interactions.
- 3D Translation: Translating predicted 2D trajectories to precise and safe 3D robot movements in physical environments is a challenge not fully addressed by the current evaluation metrics.
- Future Directions: The authors suggest that future work could explore incorporating larger-scale, more comprehensive expert datasets to address complex real-world challenges. They also point to advancements in 3D shape understanding and reconstruction as potential avenues for directly translating 2D projections into real-world 3D applications. Incorporating external commonsense knowledge bases or self-monitoring modules is also suggested to improve robustness in ambiguous scenarios.
Summary updated