Abstract
GMAP is a comprehensive "perception-to-interaction" framework designed to enable robots to manipulate articulated objects such as doors, drawers, and laptops. While high-level reasoning is increasingly handled by Large Language Models, precise physical coordination remains a significant challenge. By combining a multi-scale pre-trained point cloud encoder with precise joint estimation and affordance mapping, GMAP allows robots to plan and execute complex trajectories from simple high-level commands. Evaluated on PartNet-Mobility and in SAPIEN simulations, the framework achieves state-of-the-art performance in part segmentation and manipulation success rates, demonstrating robust generalization across unseen object categories.
Introduction
Articulated objects present a unique challenge for robotics due to their specific kinematic constraints—for example, a door swings on a hinge, while a drawer slides on a rail. While Large Language Models (LLMs) excel at high-level task planning (the "brain"), robots often lack the "cerebellum" required for precise physical interaction. Existing research typically addresses part segmentation, joint axis estimation, and affordance prediction in isolation. GMAP aims to bridge these gaps into a unified system that generalizes across different object categories even when annotated data is scarce.
Methods
The GMAP framework utilizes a multi-stage pipeline to transform raw point clouds into actionable manipulation plans.
Multi-Scale Feature Extraction (MSFE)
To handle objects of various sizes, the system partitions point clouds into multiple scales and employs a Vision Transformer (ViT) encoder. This module is pre-trained using self-supervised learning (Point-MGE) on the ShapeNet dataset, allowing the model to learn general 3D geometric features before being fine-tuned for articulated tasks.
Specialized Downstream Networks
- Seg-Net: Performs part-level segmentation and identifies movable components.
- Para-Net: Estimates kinematic parameters, including joint type (revolute vs. prismatic), orientation, position, and the current state of the object.
- Afford-Net: Predicts action proposals (where to grab) and action scores (likelihood of success).
Dynamic Trajectory Planning
Unlike baseline methods that rely on static motion primitives, GMAP uses estimated joint parameters to calculate precise trajectories in SE(3) space. This ensures the robot's end-effector follows the natural kinematic arc or line of the object's movement.
Experimental Evaluation
The authors utilized several key benchmarks for training and validation:
- ShapeNet: Used for large-scale self-supervised pre-training (50,000+ models).
- PartNet-Mobility: The primary dataset for evaluating articulated part segmentation and joint estimation.
- SAPIEN: A physics simulator used to test manipulation success rates.
Results and Key Insights
GMAP achieved state-of-the-art (SOTA) results across several metrics. In perception tasks, it reached an 80% Intersection over Union (IoU) in part segmentation and reduced joint orientation error to just 0.42°. In the SAPIEN simulator, GMAP reached a 36.94% success rate for instruction-based pushing, significantly outperforming previous benchmarks like AdaAfford (27.71%).
Key insights from the study include:
- Pre-training Efficacy: Pre-training on general 3D shapes significantly boosts performance on specialized articulated tasks by providing a better understanding of 3D structure.
- Multi-Scale Necessity: Capturing both local features (like hinges) and global features (like doors) is vital for precise joint estimation.
- Generalization: When trained on cabinets and tested on unseen categories like ovens and fridges, the model maintained high accuracy, proving it learned general principles of articulation.
Conclusion and Future Work
While GMAP demonstrates robust performance, the authors identified areas for future refinement, including addressing segmentation artifacts in tightly packed objects and improving the accuracy of initial interaction points. Future work will also explore multi-joint articulated objects, such as folding lamps or multi-link robotic arms, to further expand the framework's versatility.
Summary updated