CheckManual: A New Challenge and Benchmark for Manual-based Appliance Manipulation

Yuxing Long, Jiyao Zhang, Mingjie Pan, Tianshu Wu, Taewhan Kim, Hao Dong

Introduction

While modern robotics has made significant strides in manipulating simple objects like hammers or kitchen utensils through visual common sense, household appliances—such as microwaves, dishwashers, and coffee makers—remain a formidable challenge. The primary difficulty lies in the arbitrary nature of their interfaces: two identical-looking buttons on different microwave models might serve entirely different functions, such as setting a timer versus adjusting power levels. To operate these machines correctly, a robot must perform a task familiar to humans: consulting the instruction manual.

CheckManual is a novel benchmark and dataset designed to bridge the gap between Natural Language Processing (NLP) and physical robotics. By requiring robots to comprehend multi-page PDF manuals to plan and execute tasks, CheckManual moves beyond simple visual recognition toward knowledge-grounded manipulation.

The CheckManual Dataset and Generation Pipeline

A significant hurdle in training robots to read manuals is the lack of large-scale, physically grounded instructional data. Real-world manuals are often protected by copyright and lack direct alignment with 3D simulation environments. To address this, the researchers developed an LLM-assisted, human-verified pipeline to generate synthetic yet realistic manuals.

Starting with 182 CAD models from the PartNet-Mobility dataset, the team utilized Multimodal Large Language Models (MLLMs) to annotate parts and functions. These annotations were then used to programmatically generate 1,107 professional-grade, multi-page manuals using LaTeX. Each manual includes:

  • Detailed diagrams and control panel close-ups.
  • Step-by-step operating instructions.
  • Safety warnings and technical specifications.

The resulting dataset covers 369 appliance instances across 11 categories, featuring 2,211 manipulable parts and 1,464 specific manipulation tasks. These tasks range from simple single-step actions to complex, long-horizon sequences involving up to 18 steps.

ManualPlan: A Baseline for Knowledge-Grounded Manipulation

To establish a baseline for this new challenge, the authors proposed ManualPlan, a modular framework designed to parse manuals and execute physical actions. ManualPlan consists of three core components:

  1. Manual Resolution: This module uses Optical Character Recognition (OCR) and MLLMs to extract and structure text and visual elements from the PDF manual.
  2. Manipulation Planning: A Large Language Model (LLM) processes the extracted information to generate a high-level sequence of actions (e.g., "Open Door," "Rotate Knob 90°") tailored to the specific appliance and task.
  3. Part Alignment: To bridge the gap between the manual's text and the robot's visual input, this module employs a "Set-of-Mark" (SoM) approach. It combines Grounding-DINO and Segment Anything (SAM) to map textual descriptions to specific visual masks in the robot’s camera feed.

Benchmarking and Results

The CheckManual benchmark is organized into three tracks of increasing difficulty, ranging from Track 1 (perfectly aligned CAD models) to Track 3 (pure manual-based manipulation where the robot only has RGB-D input and the PDF).

The "Manual Gap"

The experiments revealed a clear "Manual Gap." In Track 1, the ManualPlan model achieved a 20.7% success rate. However, when the manual was removed from the input (the "w/o manual" ablation), the success rate plummeted to 7.99%. This confirms that visual appearance alone is insufficient for disambiguating part functions and that the manual serves as a critical "knowledge base" for the robot.

Challenges in Execution

In the most realistic setting (Track 3), the model achieved a 5.18% task completion rate. While this figure is low, it highlights the immense difficulty of the task and establishes a challenging baseline for future research. The primary bottleneck identified was error accumulation in long-horizon tasks. A single mistake—such as misidentifying a button or misinterpreting a step in a 10-step sequence—leads to total task failure.

Key Insights and Future Directions

The CheckManual project underscores that appliance manipulation is as much a knowledge-retrieval problem as it is a vision problem. By treating the manual as a structured source of truth, researchers can enable "zero-shot" capabilities, allowing robots to interact with unfamiliar appliances provided they have access to the documentation.

However, several limitations remain. Current MLLMs struggle to perfectly map 2D manual diagrams to 3D physical objects, particularly when interfaces feature multiple similar-looking buttons. Furthermore, while the researchers demonstrated a proof-of-concept deployment on a Franka Panda robot, the transition from simulation to the noisy, visually complex real world remains a significant hurdle. Future work will likely focus on improving multi-modal reasoning and developing more robust error-recovery mechanisms to handle the complexities of real-world instructional logic.

Summary updated

BibTeX
@inproceedings{long2025checkmanual,
  author = {Yuxing Long and Jiyao Zhang and Mingjie Pan and Tianshu Wu and Taewhan Kim and Hao Dong},
  title = {{CheckManual: A New Challenge and Benchmark for Manual-based Appliance Manipulation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.02104},
}
Figure