TL;DR
Robi Butler is a multimodal system that allows users to "Zoom-call" a household robot to perform chores remotely. By combining Large Language Models (LLMs) for task planning with Vision-Language Models (VLMs) for spatial grounding, the system enables users to give natural commands using a mix of voice, text, and hand-pointing gestures to execute complex, zero-shot manipulation tasks in a real home environment.
Introduction
Remote Human-Robot Interaction (HRI) has historically faced a difficult choice: high-latency teleoperation, which is mentally exhausting for the user, or rigid, pre-programmed commands that fail in the messy, unpredictable environment of a real home.
Language is naturally ambiguous—telling a robot to "pick up the bottle" fails if there are multiple bottles on the table. While humans solve this by pointing, translating a remote user's "click" on a 2D video screen into a 3D physical action for a robot is technically challenging. Robi Butler bridges this gap by creating a "clickable world" interface that allows remote users to guide robots naturally, reducing cognitive load while maintaining the flexibility of natural language.
Methods
The Robi Butler architecture consists of three main pillars:
- Communication Interface: The system leverages Zoom for bidirectional video, audio, and text. To handle gestures, a custom Flask-based "gesture website" creates a "clickable world" where a user’s point on a video frame is transmitted to the robot as a coordinate.
- High-Level Behavior Module: Powered by GPT-4o, this module acts as the "brain." It interprets multimodal instructions (e.g., "Bring this [gesture] to the sofa").
- Primitive Skills (The "Body"): The robot uses a suite of foundation models (like OWLv2, SAM, and Contact-GraspNet) to turn high-level plans into physical movement, including 6-DOF grasp poses for unseen objects.
Key Insights and Ideas
- The Power of the Placeholder: By teaching the LLM to use a
*symbol in its generated code whenever a demonstrative pronoun (like "this") is used, the authors decoupled high-level reasoning from low-level perception. This allows the robot to "wait" for a gesture to resolve ambiguity without needing to re-plan the entire task. - Multimodal Synergy: User studies revealed that voice and gestures are complementary. Voice is excellent for expressing intent and complex logic, while gestures provide the "spatial precision" that language lacks.
Key Results
- High Reliability: Robi Butler achieved a 96.7% task success rate across real-world household chores. Notably, the Planning Success Rate was 100%, meaning the LLM never failed to generate a logical sequence of actions.
- Efficiency: On average, tasks were completed in 105 seconds with only 2.3 interactions (1.5 voice commands + 0.8 gestures) per task.
- User Experience: In a study with 12 participants, the multimodal system significantly outperformed unimodal baselines, receiving a System Usability Scale (SUS) score of 75.83 (rated as "Good").
Conclusion and Future Work
While Robi Butler demonstrates the potential for remote household assistance, execution bottlenecks remain, such as VLM misidentification of similar objects. Future work will explore incorporating eye-gaze tracking to reduce physical fatigue from gestures and integrating tactile-language models to handle delicate force control for complex physics tasks.
Summary updated