A Mobile Manipulation System for One-Shot Teaching of Complex Tasks in Homes

Max Bajracharya, James Borders, Daniel M. Helmick, Thomas Kollar, Michael Laskey, John Leichty, Jeremy Ma, Umashankar Nagarajan, Akiyoshi Ochiai, Josh Petersen, Krishna Shankar, Kevin Stone, Yutaka Takaoka

Abstract

We describe a mobile manipulation hardware and software system capable of autonomously performing complex human-level tasks in real homes, after being taught the task with a single demonstration from a person in virtual reality. This is enabled by a highly capable mobile manipulation robot, whole-body task space hybrid position/force control, teaching of parameterized primitives linked to a robust learned dense visual embeddings representation of the scene, and a task graph of the taught behaviors. We demonstrate the robustness of the approach by presenting results for performing a variety of tasks, under different environmental conditions, in multiple real homes. Our approach achieves an 85% overall success rate on three tasks that consist of an average of 45 behaviors each.

Introduction

Robots in domestic environments face a significant "long tail" problem: every home features a unique layout, different objects, and specific user requirements. Traditional robotics approaches typically struggle with this variability. Hard-coded programming is too brittle for the diversity of home environments, while large-scale Reinforcement Learning (RL) requires thousands of trials that are impractical to perform in a physical kitchen.

Researchers at the Toyota Research Institute (TRI) propose a shift in philosophy: specialization over generalization. Rather than building a robot that knows how to clean every kitchen, they have developed a system that a non-expert can quickly specialize to their specific home using a single demonstration.

System Architecture and Methods

The system architecture integrates high-level human intuition with low-level robotic precision through a "teach-and-repeat" framework.

The system architecture integrates audio, task graphs, and dense visual embeddings to drive robot behaviors and control.

Hardware and Control

The platform is a custom 31-degree-of-freedom (DOF) mobile robot featuring a pseudo-holonomic base, a 5-DOF torso, and dual 7-DOF arms. To manage this complexity, the robot utilizes a Quadratic Programming (QP) solver for whole-body control, ensuring collision avoidance and balance while executing tasks. Perception is handled by a wide-field-of-view head equipped with four depth cameras and two high-resolution stereo cameras.

VR Teaching and Task Representation

A human operator uses an HTC Vive VR headset to provide a single demonstration. By seeing through the robot's cameras, the operator places "keyframes" and defines parameterized behaviors such as grasping, pulling, or wiping.

The VR teaching interface allows an operator to provide demonstrations and define task parameters in a 3D point cloud environment.

Tasks are represented as directed task graphs. Each node represents a behavior, and edges represent success or failure conditions. This structure allows for "recoverable failures"; for instance, if the robot misses a handle, the graph can loop back to attempt the grasp again rather than failing the entire mission.

Dense Visual Embeddings

To handle environmental changes like lighting or slight object shifts without CAD models, the system employs dense visual embeddings. Using a deep neural network (ResNet-101 + Feature Pyramid Network), every pixel is mapped to a high-dimensional embedding space. This acts as "visual glue," allowing the robot to find correspondences between the demonstration and the current scene autonomously.

Experimental Results

The system was evaluated in two real-world homes on three complex tasks: fetching a bottle from a refrigerator, retrieving a cup from a dishwasher, and moving multiple objects based on voice commands.

Across 60 autonomous trials, the system achieved an 85% overall success rate. Remarkably, the behaviors were found to be 99.6% reliable in terms of resulting in either a success or a recoverable failure. While highly robust, the system currently operates significantly slower than a human, averaging approximately 20x slower execution speeds.

Discussion and Future Work

By embracing specialization, this system demonstrates that complex household tasks can be automated with minimal human intervention. However, limitations remain. The current teaching process still requires manual annotation in VR, and the system can be sensitive to major discrete state changes (e.g., a door being half-open when it was taught as closed). Future research will focus on reducing teaching overhead and improving the robot's ability to generalize tasks across different homes with even less human guidance.

Summary updated

Abstract

We describe a mobile manipulation hardware and software system capable of autonomously performing complex human-level tasks in real homes, after being taught the task with a single demonstration from a person in virtual reality. This is enabled by a highly capable mobile manipulation robot, whole-body task space hybrid position/force control, teaching of parameterized primitives linked to a robust learned dense visual embeddings representation of the scene, and a task graph of the taught behaviors. We demonstrate the robustness of the approach by presenting results for performing a variety of tasks, under different environmental conditions, in multiple real homes. Our approach achieves 85% overall success rate on three tasks that consist of an average of 45 behaviors each.
BibTeX
@inproceedings{bajracharya2020mobile,
  author = {Max Bajracharya and James Borders and Daniel M. Helmick and Thomas Kollar and Michael Laskey and John Leichty and Jeremy Ma and Umashankar Nagarajan and Akiyoshi Ochiai and Josh Petersen and Krishna Shankar and Kevin Stone and Yutaka Takaoka},
  title = {{A Mobile Manipulation System for One-Shot Teaching of Complex Tasks in Homes}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2020},
  doi = {10.1109/icra40945.2020.9196677},
}
Figure