TL;DR
Robi Butler is a multimodal tele-assistant system that lets remote users direct a household mobile manipulator using natural language, voice, and web-based pointing gestures. By combining Large Language Models (GPT-4) for high-level planning and Vision-Language Models (SAM, GPT-4V, Contact-GraspNet) for open-vocabulary execution, the system bridges the gap between ambiguous spoken instructions and physical real-world manipulation.
What Problem Does This Paper Address?
Instructing a household robot while away from home (e.g., checking refrigerator contents or heating food) is inherently difficult. Relying solely on text or voice forces users into elaborate, unnatural descriptions to disambiguate target objects (e.g., "the mug slightly to the left of the blue bowl"). Conversely, purely gesture-based interfaces ignore rich contextual language and often rely on fixed vocabulary or rigid third-person camera views.
Robi Butler addresses this by enabling bidirectional, multimodal remote communication. It gives remote users a simple way to combine speech and deictic (pointing) gestures in real time to disambiguate targets, query the environment, and execute complex household manipulation tasks zero-shot.
Methods
The Robi Butler architecture consists of three interconnected modules:
[Remote User: AR/Web]
└── (Voice, Text, Pointing) ──> Communication Interface (Zoom + Flask Web App)
│
▼
High-Level Behavior Module (GPT-4 Planner)
│
▼
Fundamental Skills (ROS, SAM, Contact-GraspNet, IL)
- Communication Interfaces:
- A standard Zoom web interface captures video, text, and voice (converted to text via live transcription).
- A custom Flask-based gesture web portal streams robot video (5 Hz) to the user's browser, allowing them to click/point on objects directly on-screen.
- High-Level Behavior Module (Task Planner):
- Driven by GPT-4, this module interprets user instructions and maps deictic references (like the word "this") into wildcards
*in a structured plan (e.g.,pick(*), placeon('plate')). - An execution manager matches these wildcards to the user’s recent click coordinates and coordinates human-in-the-loop clarification dialog when ambiguity arises (e.g., asking "Which one are you referring to?" if multiple candidates are detected).
- Fundamental Skills:
- Open-Vocabulary Pick & Place: Combines open-vocabulary object detection with Segment Anything (SAM) and Contact-GraspNet for 6-DoF grasp synthesis, planned via MoveIt.
- Articulated Interaction: Uses imitation learning (~30 teleoperated demonstrations via VR controller) to open refrigerators and microwaves.
- Visual Question Answering (VQA): Leverages GPT-4V to answer open-ended queries either by planning multi-step visual inspections or by grounding questions on user-selected regions using visual prompts (masks/points from SAM).
Datasets
- Training/Pretraining: Robi Butler operates largely zero-shot using off-the-shelf foundation models (GPT-4, GPT-4V, SAM, and Contact-GraspNet). For articulated opening skills (fridge, microwave), the authors collected an internal dataset averaging 30 VR teleoperation demonstrations per action.
- Evaluation Tasks: The authors designed real-world benchmark tasks inspired by the American Time Use Survey (ATUS), covering categories like food and drink preparation, interior cleaning, organization, and care services.
Key Insights and Ideas
- Deictic Grounding via Wildcard Code Generation: Instead of forcing the LLM to process raw coordinate streams, the planner cleanly separates high-level logic from spatial pointing by emitting placeholder tokens (
*), which are dynamically bound to click events by the execution manager. - 100% Planning vs. Physical Execution Gap: While the LLM exhibited a 100% success rate in translating user intent into valid action sequences, overall task failures stemmed strictly from physical execution and human interface mishaps.
- Interactive VQA Over Passive Sensing: For queries requiring visual inspection (e.g., checking if food in the fridge is spoiled), the system does not passively query a stationary camera; it coordinates mobile navigation and manipulator movement before querying GPT-4V.
Key Results
The authors evaluated Robi Butler across five representative real-world tasks (3 trials each, 15 trials total) performed by a user in a separate room:
- Task Success Rate: 80% (12/15 tasks successfully completed within 5 minutes).
- Planning Success Rate: 100% across all tasks.
- Average Completion Time: 135 seconds per completed task (most time spent on navigation and arm trajectory execution).
- Interaction Overhead: Highly efficient, requiring an average of only 1.4 user interactions per task.
Limitations and Future Work
- Hardware & Motion Execution Errors: The primary sources of failure were low-level physical errors—such as the gripper accidentally bumping a cup during the grasp approach or the robot arm colliding with its own chassis while carrying a beverage.
- Accidental User Input: Users occasionally clicked the wrong pixel target on the web stream, leading to misdirected grasping.
- Limited Manipulation Repertoire: Dynamic or articulated interactions beyond opening doors (e.g., pouring, wiping, or handling deformables) are not yet supported.
- Future Directions: The authors plan to test the system across larger user studies, expand the library of low-level manipulation primitives, and improve grasp execution robustness.
Summary updated