TL;DR
ApBot is a generalizable robotic framework that enables robots to operate unfamiliar household appliances zero-shot simply by "reading" their user manuals. By combining vision-language models (VLMs) with structured symbolic state machines and closed-loop visual feedback, ApBot achieves robust, multi-step execution where end-to-end foundation models typically fail.
What Problem Does This Paper Address?
For assistive home robots to be truly useful, they must be able to operate everyday household appliances like microwaves, washing machines, and rice cookers. However, doing this in a zero-shot manner for a novel appliance is incredibly difficult.
User manuals are highly unstructured documents containing a mix of text, diagrams, and symbols. Furthermore, appliances operate under strict, state-dependent workflows that are highly unforgiving of errors (e.g., you cannot set a cooking timer if the machine isn't in the correct mode). While modern large vision-language models (VLMs) possess broad semantic knowledge, using them directly as end-to-end control policies often leads to compounding errors, hallucinations, and poor fine-grained visual grounding over long operational horizons.
Methods
ApBot tackles these challenges by avoiding direct end-to-end control, opting instead to build an intermediate structured representation of the appliance.
- Structured Appliance Modeling: ApBot uses an LVLM (GPT-4o) to parse the unstructured user manual into a symbolic finite state machine. This model explicitly defines the appliance's state space (e.g., temperature, current mode), valid actions, and transition rules.
- Action Grounding: To execute these symbolic actions physically, ApBot implements a parallel perception pipeline. It combines Segment Anything (SAM) for visual region segmentation, OWL-ViT2 for open-vocabulary button/dial detection, and EasyOCR for text label extraction. A VLM then maps the manual's symbolic actions to these detected physical control elements.
- Macro Actions: To prevent long-horizon planning failures, ApBot abstracts low-level actions into code-backed "macro actions" (e.g., calculating exactly how many times to press a "+" button to reach a target temperature), significantly shortening the reasoning horizon.
- Closed-Loop Calibration: If the appliance behaves unexpectedly during execution, ApBot uses visual feedback to track the state mismatch. It then initiates an online exploration sequence to diagnose the error and update its internal transition model dynamically.
Datasets
The authors introduced a newly constructed simulated benchmark to evaluate appliance operation. It features:
- 6 appliance types: Dehumidifiers, bottle washers, rice cookers, microwave ovens, bread makers, and washing machines.
- 30 interactive instances derived directly from real-world user manuals.
- 300 goal-directed natural language instructions (10 per appliance), scaled systematically by complexity (ranging from 1 to 6 adjustable variables).
For physical validation, the system was deployed using a Kinova Gen3 robotic arm to operate three real-world appliances: a blender, an induction cooker, and a water dispenser.
Key Insights and Ideas
- Structure Beats Raw Scale: Forcing a VLM to generate a symbolic state machine rather than direct actions bridges the gap between unstructured text and precise execution. It proves that structured internal representations are vital for complex, rule-governed robotic tasks.
- Deterministic Code as an Engine: Instead of relying on a VLM to guess multi-step action sequences (like counting button clicks), ApBot delegates sequence generation to deterministic code blocks triggered by the state machine, eliminating a major source of VLM math hallucination.
- Resilience via Online Learning: Manuals can be ambiguous, and VLMs make mistakes. By treating the initial parsed model as a flexible prior that can be corrected via closed-loop visual exploration, the robot can recover gracefully from its own misinterpretations.
Key Results
- Superior Success Rates: ApBot consistently and significantly outperformed pure VLM-as-policy baselines across all appliance types in the benchmark.
- Robustness to Complexity: As tasks scaled in difficulty (from 1 variable in dehumidifiers to 6 variables in washing machines), baseline performance plummeted due to compounding reasoning errors. ApBot maintained a high, stable success rate.
- Grounding Advantages: The explicit text-and-icon grounding pipeline boosted overall performance by 18% compared to direct image-based policy reasoning, outperforming state-of-the-art general visual grounding models like Molmo.
Limitations and Future Work
- No Touchscreen Support: ApBot is currently limited to physical buttons and dials and cannot interact with modern capacitive touchscreens.
- Lack of Haptic Feedback: The system relies entirely on vision; it does not incorporate tactile feedback, which is crucial for confirming physical button clicks or managing mechanical resistance.
- Simple Manipulation Primitives: The framework only handles pressing, holding, and turning. It cannot execute more complex physical interactions like opening/closing appliance doors, plugging in power cords, or inserting/removing items from containers.
Summary updated