TL;DR
This paper introduces "Robi Butler," a novel household robot assistant that enables seamless remote interaction using a combination of voice, text, and hand gestures. By leveraging Large Language Models (LLMs) and vision-language models, Robi Butler can interpret complex, multimodal instructions to perform household tasks, bringing us closer to a future where we can manage our homes remotely.
What Problem Does This Paper Address?
The paper tackles the challenge of enabling intuitive and effective remote control of household robots. Current systems often rely on predefined commands or complex interfaces, making them difficult for everyday users. The authors envision a future where users can interact with robots as easily as they would with a human assistant, using familiar tools like video calls and natural language, augmented by gestures for precision. The core problem is how to bridge the gap between human intent expressed through multimodal inputs (voice, text, gestures) and the robot's ability to understand and execute these instructions in a real-world home environment, especially for open-ended tasks.
Methods
Robi Butler is built upon three key components:
- Communication Interface: This allows for bidirectional communication between the remote user and the robot. It integrates a Zoom platform for voice, text, and video, and a separate gesture website. Users can point to objects in the robot's first-person view, and these gestures are transmitted to the robot. This interface is designed to be latency-tolerant, enabling interaction from anywhere.
- High-level Behavior Module: This is the "brain" of Robi Butler, powered by a Large Language Model (LLM, specifically GPT-4). The LLM is prompted to act as a household robot assistant, interpreting multimodal instructions (language and gestures) to generate a sequence of executable action plans. It also handles disambiguation, for example, by using a placeholder "*" for objects referred to by "this" or "here," which is then resolved by the latest gesture selection.
- Primitive Skills: These are the robot's core capabilities, including mobile manipulation (pick, place, move, open, close) and Visual Question Answering (VQA). Crucially, many of these skills support open-vocabulary commands and can be guided by both text and pointing gestures. For manipulation, they leverage pre-trained vision-language models like OWLv2 and Segment Anything, combined with grasping models. Navigation uses standard ROS navigation stack.
Datasets
The paper does not explicitly mention training datasets for the LLM or vision-language models, implying the use of pre-trained, state-of-the-art foundation models. For evaluation, they created a set of 10 real-world household tasks (T1-T10) based on the American Time Use Survey, covering activities like food preparation, cleaning, and organization. They also conducted a user study with 12 participants using three representative tasks.
Key Insights and Ideas
- Multimodal Interaction is Key: The study strongly suggests that combining voice and gestures significantly enhances user experience, trust, and efficiency compared to unimodal interactions. Gestures provide the necessary precision that language alone often lacks.
- LLMs as General-Purpose Robot Planners: The paper demonstrates the effectiveness of using LLMs as high-level planners for complex, open-ended tasks, enabling zero-shot generalization to new instructions.
- Bridging the Gap Between Language and Action: The system's ability to ground abstract language commands and precise gestural inputs into concrete robot actions is a significant step towards more natural human-robot collaboration.
- User-Friendly Remote Control: The integration of familiar tools like Zoom and a simple gesture interface makes remote robot control accessible to a wider audience, moving beyond specialized teleoperation systems.
Key Results
- High Task Success Rate: Robi Butler achieved an average task success rate of 96.7% across 10 real-world household tasks, with a planning success rate of 100%.
- Efficient Task Completion: Tasks were completed in an average of 105 seconds, requiring a low average of 2.3 interactions (1.5 voice, 0.8 gesture) per task.
- Improved User Experience: In the user study, Robi Butler (Gesture + Voice) outperformed unimodal systems in terms of trust (5.77 vs. 4.71-4.98) and usability (SUS score of 75.83 vs. 47.29-70.42). It also achieved the minimum perceived cognitive load.
- Gesture-Only and Robi Butler Achieved Perfect Success: Both the gesture-only and the full Robi Butler system achieved 100% task success in the user study, highlighting the importance of gestural input for precision.
Limitations and Future Work
- Low-Level Execution Challenges: While planning was successful, slight lags in task success rate compared to planning success rate indicate challenges in low-level action execution.
- Latency: While the interface is designed to be latency-tolerant, the system still experiences delays (up to 0.2s in Singapore, 0.5s for long distances).
- User Workload with Multimodal Input: One user expressed that switching between modalities could be challenging, suggesting that further optimization to reduce physical workload (e.g., through eye gaze tracking) is desirable.
- Future Directions: The authors aim to make Robi Butler more adaptable, capable of lifelong learning, personalized interactions, and handling more complex tasks, potentially incorporating tactile feedback.
Summary updated