Robot Operation of Home Appliances by Reading User Manuals

Jian Zhang, Hanbo Zhang, Anxing Xiao, David Hsu

TL;DR

ApBot is a generalizable robotic framework that enables robots to operate unfamiliar household appliances zero-shot simply by "reading" their user manuals. By combining vision-language models (VLMs) with structured symbolic state machines and closed-loop visual feedback, ApBot achieves robust, multi-step execution where end-to-end foundation models typically fail.

What Problem Does This Paper Address?

For assistive home robots to be truly useful, they must be able to operate everyday household appliances like microwaves, washing machines, and rice cookers. However, doing this in a zero-shot manner for a novel appliance is incredibly difficult.

User manuals are highly unstructured documents containing a mix of text, diagrams, and symbols. Furthermore, appliances operate under strict, state-dependent workflows that are highly unforgiving of errors (e.g., you cannot set a cooking timer if the machine isn't in the correct mode). While modern large vision-language models (VLMs) possess broad semantic knowledge, using them directly as end-to-end control policies often leads to compounding errors, hallucinations, and poor fine-grained visual grounding over long operational horizons.

Methods

ApBot tackles these challenges by avoiding direct end-to-end control, opting instead to build an intermediate structured representation of the appliance.

Figure 2: Overview of ApBot. The key of ApBot is the appliance model built from a textual manual, based on which we generate actions for novel appliance operation. To handle possible model errors, we calibrate the model with observable feedback during execution in a close-loop manner.
  1. Structured Appliance Modeling: ApBot uses an LVLM (GPT-4o) to parse the unstructured user manual into a symbolic finite state machine. This model explicitly defines the appliance's state space (e.g., temperature, current mode), valid actions, and transition rules.
  2. Action Grounding: To execute these symbolic actions physically, ApBot implements a parallel perception pipeline. It combines Segment Anything (SAM) for visual region segmentation, OWL-ViT2 for open-vocabulary button/dial detection, and EasyOCR for text label extraction. A VLM then maps the manual's symbolic actions to these detected physical control elements.
  3. Macro Actions: To prevent long-horizon planning failures, ApBot abstracts low-level actions into code-backed "macro actions" (e.g., calculating exactly how many times to press a "+" button to reach a target temperature), significantly shortening the reasoning horizon.
  4. Closed-Loop Calibration: If the appliance behaves unexpectedly during execution, ApBot uses visual feedback to track the state mismatch. It then initiates an online exploration sequence to diagnose the error and update its internal transition model dynamically.
Figure 3: Overview of action grounding with visual observations.

Datasets

The authors introduced a newly constructed simulated benchmark to evaluate appliance operation. It features:

  • 6 appliance types: Dehumidifiers, bottle washers, rice cookers, microwave ovens, bread makers, and washing machines.
  • 30 interactive instances derived directly from real-world user manuals.
  • 300 goal-directed natural language instructions (10 per appliance), scaled systematically by complexity (ranging from 1 to 6 adjustable variables).

For physical validation, the system was deployed using a Kinova Gen3 robotic arm to operate three real-world appliances: a blender, an induction cooker, and a water dispenser.

Key Insights and Ideas

  • Structure Beats Raw Scale: Forcing a VLM to generate a symbolic state machine rather than direct actions bridges the gap between unstructured text and precise execution. It proves that structured internal representations are vital for complex, rule-governed robotic tasks.
  • Deterministic Code as an Engine: Instead of relying on a VLM to guess multi-step action sequences (like counting button clicks), ApBot delegates sequence generation to deterministic code blocks triggered by the state machine, eliminating a major source of VLM math hallucination.
  • Resilience via Online Learning: Manuals can be ambiguous, and VLMs make mistakes. By treating the initial parsed model as a flexible prior that can be corrected via closed-loop visual exploration, the robot can recover gracefully from its own misinterpretations.

Key Results

  • Superior Success Rates: ApBot consistently and significantly outperformed pure VLM-as-policy baselines across all appliance types in the benchmark.
  • Robustness to Complexity: As tasks scaled in difficulty (from 1 variable in dehumidifiers to 6 variables in washing machines), baseline performance plummeted due to compounding reasoning errors. ApBot maintained a high, stable success rate.
  • Grounding Advantages: The explicit text-and-icon grounding pipeline boosted overall performance by 18% compared to direct image-based policy reasoning, outperforming state-of-the-art general visual grounding models like Molmo.

Limitations and Future Work

  • No Touchscreen Support: ApBot is currently limited to physical buttons and dials and cannot interact with modern capacitive touchscreens.
  • Lack of Haptic Feedback: The system relies entirely on vision; it does not incorporate tactile feedback, which is crucial for confirming physical button clicks or managing mechanical resistance.
  • Simple Manipulation Primitives: The framework only handles pressing, holding, and turning. It cannot execute more complex physical interactions like opening/closing appliance doors, plugging in power cords, or inserting/removing items from containers.

Summary updated

Abstract

Operating home appliances, among the most common tools in every household, is a critical capability for assistive home robots. This paper presents ApBot, a robot system that operates novel household appliances by "reading" their user manuals. ApBot faces multiple challenges: (i) infer goal-conditioned partial policies from their unstructured, textual descriptions in a user manual document, (ii) ground the policies to the appliance in the physical world, and (iii) execute the policies reliably over potentially many steps, despite compounding errors. To tackle these challenges, ApBot constructs a structured, symbolic model of an appliance from its manual, with the help of a large vision-language model (VLM). It grounds the symbolic actions visually to control panel elements. Finally, ApBot closes the loop by updating the model based on visual feedback. Our experiments show that across a wide range of simulated and real-world appliances, ApBot achieves consistent and statistically significant improvements in task success rate, compared with state-of-the-art large VLMs used directly as control policies. These results suggest that a structured internal representations plays an important role in robust robot operation of home appliances, especially, complex ones.
BibTeX
@inproceedings{zhang2025robot,
  author = {Jian Zhang and Hanbo Zhang and Anxing Xiao and David Hsu},
  title = {{Robot Operation of Home Appliances by Reading User Manuals}},
  booktitle = {Conference on Robot Learning},
  year = {2025},
  url = {https://openreview.net/forum?id=wZUQq0JaL6},
}
Figure