Manual2Skill: Learning to Read Manuals and Acquire Robotic Skills for Furniture Assembly Using Vision-Language Models

Chenrui Tie, Shengxiang Sun, Jinxuan Zhu, Yiwei Liu, Jingxiang Guo, Yue Hu, Haonan Chen, Junting Chen, Ruihai Wu, Lin Shao

TL;DR

Manual2Skill is a new framework that enables robots to assemble furniture by "reading" human-designed instruction manuals. By combining the high-level reasoning of Vision-Language Models (VLMs) with specialized 6D pose estimation and motion planning, the system can autonomously transform abstract sketches into a sequence of precise robotic manipulations.

What Problem Does This Paper Address?

Assembling IKEA furniture is a rite of passage for humans, but a nightmare for robots. While humans can easily map abstract 2D sketches to 3D physical parts, robots typically require exhaustive, step-by-step demonstrations or massive amounts of reinforcement learning data to perform similar long-horizon tasks.

The core challenge lies in the "semantic gap": instruction manuals are designed for human intuition, using symbols and simplified diagrams that don't translate directly to the coordinate systems and joint angles robots use. Previous robotic assembly research often focused on either high-level planning or low-level control in isolation. Manual2Skill aims to bridge this gap, creating an end-to-end pipeline that moves from a stack of parts and a PDF manual to a completed piece of furniture.

Methods

The Manual2Skill framework consists of three main stages, as illustrated in the overview:

Fig. 1: Overview of Manual2Skill Framework. We propose Manual2Skill, which learns manipulation skills from manuals, enabling robots to understand and execute complex manipulation tasks in a manner akin to humans. The green region showcases the input of our pipeline: the pictures of the assembly manual and real parts. The blue region depicts our pipeline: 1) a Vision- Language Model (VLM) generates a Hierarchical Assembly Graph, 2) a per-step pose estimation module predicts the 6D-poses of components, and 3) a motion planning and execution module controls the robot arms to assemble the furniture autonomously.
  1. VLM-Guided Hierarchical Graph Generation: The system uses a VLM (specifically GPT-4o) to interpret the manual. Instead of a simple list of steps, it generates a Hierarchical Assembly Graph (HAG). This graph defines leaf nodes as atomic parts and parent nodes as subassemblies, capturing the dependencies and parallelizable steps of the assembly process. This process is split into two stages: first associating real-world parts with roles in the manual, and then identifying which parts are involved in each specific manual page.
  2. Per-step Assembly Pose Estimation: Once the "what" and "when" are decided, the robot needs to know the "where." A specialized neural network takes the manual image and the point clouds of the involved parts to predict their relative 6D poses. This module uses a Graph Neural Network (GNN) to fuse multi-modal information, capturing the spatial relationships between components depicted in the manual.
  3. Motion Planning and Execution: Finally, the system maps these predicted poses into the robot's world frame. It uses RRT-Connect for collision-free path planning and employs heuristic grasping strategies tailored to the shape of the components (e.g., stick-shaped vs. flat-shaped).
Fig. 2: Framework Overview. (1) GPT-4o [1] is queried with manual pages to generate a sequential assembly plan, represented as a hierarchical assembly graph. (2) The furniture components’ point clouds and corresponding manual images are processed by a pose estimation module to predict target poses for each component. (3) The system sequentially executes the assembly by planning and performing robotic actions based on the hierarchical assembly graph and estimated poses.

Datasets

The authors evaluated their framework using several benchmarks:

  • IKEA-Manuals Dataset: Used to test the VLM's ability to generate accurate assembly graphs across 102 different furniture items.
  • New Pose Estimation Dataset: The authors created a large-scale dataset using PartNet (chairs, tables, lamps) consisting of 12,000 training and 5,200 testing samples per category to train their 6D pose regressor.
  • Real-World IKEA Items: The system was validated on four physical items: the Flisat stool, Variera shelf, Sundvik chair, and Knagglig box.

Key Insights and Ideas

  • Decoupling Reasoning from Geometry: VLMs are excellent at semantic reasoning (understanding that a leg goes under a seat) but poor at precise spatial geometry. Manual2Skill succeeds by letting the VLM handle the high-level logic (the graph) while a specialized geometric model handles the precise alignment (pose estimation).
  • The Power of Hierarchy: Representing assembly as a tree rather than a linear list allows the robot to understand subassemblies. This is crucial for complex items where multiple parts must be joined before being attached to the main body.
  • Manual Segmentation is Critical: The authors found that feeding a VLM an entire manual at once leads to "distraction" from irrelevant elements like screws or human figures. Providing cropped, per-step images significantly improved the success rate of graph generation.
  • Handling Equivalent Parts: The framework explicitly accounts for interchangeable parts (like four identical chair legs). The pose estimation loss function is designed to be permutation-invariant for these parts, preventing the model from getting confused by identical geometries.

Key Results

Manual2Skill demonstrated significant improvements over heuristic and single-image baselines:

  • Graph Generation: For furniture with 5-6 parts, Manual2Skill achieved a 55.6% success rate, whereas previous heuristic methods like GeoCluster struggled at 2.8%.
  • Simulation: In PyBullet environments, the framework successfully assembled 29 out of 50 furniture items (58% success rate), nearly doubling the performance of the baseline (30%).
  • Real-World Execution: The system achieved high Average Completion Rates (ACR) across four different IKEA items, outperforming the Mean-Max Pool baseline in every category. For example, it achieved an 85% success rate on the Knagglig box in real-world trials.

Limitations and Future Work

While a major step forward, the authors note several areas for improvement:

  • Small Parts: The current system focuses on major components and overlooks small fasteners like screws and nails.
  • Contact-Rich Insertion: The final "mating" or insertion of parts is currently performed with human assistance or open-loop control. Future work aims to integrate force and tactile sensing for fully autonomous, closed-loop assembly.
  • Complexity Scaling: Performance still drops as the number of parts increases beyond 7-9, suggesting that even more robust VLM prompting or hierarchical decomposition strategies are needed for very complex assemblies.

Summary updated

Abstract

Humans possess an extraordinary ability to understand and execute complex manipulation tasks by interpreting abstract instruction manuals. For robots, however, this capability remains a substantial challenge, as they cannot interpret abstract instructions and translate them into executable actions. In this paper, we present Manual2Skill, a novel framework that enables robots to perform complex assembly tasks guided by high-level manual instructions. Our approach leverages a Vision-Language Model (VLM) to extract structured information from instructional images and then uses this information to construct hierarchical assembly graphs. These graphs represent parts, subassemblies, and the relationships between them. To facilitate task execution, a pose estimation model predicts the relative 6D poses of components at each assembly step. At the same time, a motion planning module generates actionable sequences for real-world robotic implementation. We demonstrate the effectiveness of Manual2Skill by successfully assembling several real-world IKEA furniture items. This application highlights its ability to manage long-horizon manipulation tasks with both efficiency and precision, significantly enhancing the practicality of robot learning from instruction manuals. This work marks a step forward in advancing robotic systems capable of understanding and executing complex manipulation tasks in a manner akin to human capabilities.Project Page: https://owensun2004.github.io/Furniture-Assembly-Web/
BibTeX
@inproceedings{tie2025manual,
  author = {Chenrui Tie and Shengxiang Sun and Jinxuan Zhu and Yiwei Liu and Jingxiang Guo and Yue Hu and Haonan Chen and Junting Chen and Ruihai Wu and Lin Shao},
  title = {{Manual2Skill: Learning to Read Manuals and Acquire Robotic Skills for Furniture Assembly Using Vision-Language Models}},
  booktitle = {Robo-3Dvlm @ CVPR},
  year = {2025},
  url = {https://openreview.net/forum?id=K3JtdYXT6J},
}
Figure