TL;DR
Manual2Skill is a new framework that enables robots to assemble furniture by "reading" human-designed instruction manuals. By combining the high-level reasoning of Vision-Language Models (VLMs) with specialized 6D pose estimation and motion planning, the system can autonomously transform abstract sketches into a sequence of precise robotic manipulations.
What Problem Does This Paper Address?
Assembling IKEA furniture is a rite of passage for humans, but a nightmare for robots. While humans can easily map abstract 2D sketches to 3D physical parts, robots typically require exhaustive, step-by-step demonstrations or massive amounts of reinforcement learning data to perform similar long-horizon tasks.
The core challenge lies in the "semantic gap": instruction manuals are designed for human intuition, using symbols and simplified diagrams that don't translate directly to the coordinate systems and joint angles robots use. Previous robotic assembly research often focused on either high-level planning or low-level control in isolation. Manual2Skill aims to bridge this gap, creating an end-to-end pipeline that moves from a stack of parts and a PDF manual to a completed piece of furniture.
Methods
The Manual2Skill framework consists of three main stages, as illustrated in the overview:
- VLM-Guided Hierarchical Graph Generation: The system uses a VLM (specifically GPT-4o) to interpret the manual. Instead of a simple list of steps, it generates a Hierarchical Assembly Graph (HAG). This graph defines leaf nodes as atomic parts and parent nodes as subassemblies, capturing the dependencies and parallelizable steps of the assembly process. This process is split into two stages: first associating real-world parts with roles in the manual, and then identifying which parts are involved in each specific manual page.
- Per-step Assembly Pose Estimation: Once the "what" and "when" are decided, the robot needs to know the "where." A specialized neural network takes the manual image and the point clouds of the involved parts to predict their relative 6D poses. This module uses a Graph Neural Network (GNN) to fuse multi-modal information, capturing the spatial relationships between components depicted in the manual.
- Motion Planning and Execution: Finally, the system maps these predicted poses into the robot's world frame. It uses RRT-Connect for collision-free path planning and employs heuristic grasping strategies tailored to the shape of the components (e.g., stick-shaped vs. flat-shaped).
Datasets
The authors evaluated their framework using several benchmarks:
- IKEA-Manuals Dataset: Used to test the VLM's ability to generate accurate assembly graphs across 102 different furniture items.
- New Pose Estimation Dataset: The authors created a large-scale dataset using PartNet (chairs, tables, lamps) consisting of 12,000 training and 5,200 testing samples per category to train their 6D pose regressor.
- Real-World IKEA Items: The system was validated on four physical items: the Flisat stool, Variera shelf, Sundvik chair, and Knagglig box.
Key Insights and Ideas
- Decoupling Reasoning from Geometry: VLMs are excellent at semantic reasoning (understanding that a leg goes under a seat) but poor at precise spatial geometry. Manual2Skill succeeds by letting the VLM handle the high-level logic (the graph) while a specialized geometric model handles the precise alignment (pose estimation).
- The Power of Hierarchy: Representing assembly as a tree rather than a linear list allows the robot to understand subassemblies. This is crucial for complex items where multiple parts must be joined before being attached to the main body.
- Manual Segmentation is Critical: The authors found that feeding a VLM an entire manual at once leads to "distraction" from irrelevant elements like screws or human figures. Providing cropped, per-step images significantly improved the success rate of graph generation.
- Handling Equivalent Parts: The framework explicitly accounts for interchangeable parts (like four identical chair legs). The pose estimation loss function is designed to be permutation-invariant for these parts, preventing the model from getting confused by identical geometries.
Key Results
Manual2Skill demonstrated significant improvements over heuristic and single-image baselines:
- Graph Generation: For furniture with 5-6 parts, Manual2Skill achieved a 55.6% success rate, whereas previous heuristic methods like GeoCluster struggled at 2.8%.
- Simulation: In PyBullet environments, the framework successfully assembled 29 out of 50 furniture items (58% success rate), nearly doubling the performance of the baseline (30%).
- Real-World Execution: The system achieved high Average Completion Rates (ACR) across four different IKEA items, outperforming the Mean-Max Pool baseline in every category. For example, it achieved an 85% success rate on the Knagglig box in real-world trials.
Limitations and Future Work
While a major step forward, the authors note several areas for improvement:
- Small Parts: The current system focuses on major components and overlooks small fasteners like screws and nails.
- Contact-Rich Insertion: The final "mating" or insertion of parts is currently performed with human assistance or open-loop control. Future work aims to integrate force and tactile sensing for fully autonomous, closed-loop assembly.
- Complexity Scaling: Performance still drops as the number of parts increases beyond 7-9, suggesting that even more robust VLM prompting or hierarchical decomposition strategies are needed for very complex assemblies.
Summary updated