Catch & Carry: reusable neural controllers for vision-guided whole-body tasks

Josh Merel, Saran Tunyasuvunakool, Arun Ahuja, Yuval Tassa, Leonard Hasenclever, Vu Pham, Tom Erez, Greg Wayne, Nicolas HeessACM Trans. Graphics 2020 · DOI · OpenReview

TL;DR

This paper introduces a novel approach for training reusable neural controllers for humanoid robots to perform complex whole-body tasks involving object interaction, like catching and carrying. By distilling motion capture data into a general-purpose motor skill module and then training task-specific policies on top of it, they achieve robust and flexible control that can adapt to new scenarios and even be deployed in real-time.

What Problem Does This Paper Address?

Creating realistic and flexible humanoid character controllers capable of performing diverse whole-body tasks, especially those involving object interaction, is a significant challenge. Traditional methods often require manual editing or kinematic blending, which are labor-intensive and lack adaptability. While reinforcement learning (RL) has shown promise, learning complex manipulation skills from scratch is difficult due to sparse rewards and high-dimensional state spaces. This paper aims to bridge this gap by developing an integrated approach that leverages human demonstrations to build a reusable motor skill foundation, enabling more efficient and robust learning of complex tasks.

Methods

The authors propose a three-stage approach:

  1. Stage 1: Expert Policy Training: They train individual "expert" neural network policies to meticulously replicate specific motion capture clips, effectively learning to track precise trajectories.
  2. Stage 2: Skill Module Distillation (NPMP): These expert policies are then distilled into a single, reusable "Neural Probabilistic Motor Primitive" (NPMP) module. This module acts as a low-level controller that maps egocentric observations (like vision and proprioception) to motor intentions. A key innovation here is that the decoder part of the NPMP only receives egocentric information, making it generalizable across different environments.
  3. Stage 3: Task Policy Training: A high-level task policy is trained using RL to reuse the frozen NPMP module. This task policy takes task-specific inputs (like instructions or visual cues) and outputs latent "commands" that guide the NPMP. This hierarchical structure allows for efficient exploration and learning, as the NPMP constrains the search space to human-like behaviors.

The training process incorporates several key elements:

  • Egocentric Vision: The system uses first-person visual input, encouraging natural behaviors like gaze direction.
  • Curriculum Learning: Task variations (e.g., different object sizes, initial positions, task phases) are used to gradually increase difficulty and facilitate generalization.
  • Phased Task Training: For complex tasks, training is broken down into stages with simpler rewards to aid exploration.

Datasets

The primary dataset used is motion capture data collected from a single human subject performing bimanual whole-body box manipulation, ball tossing, and locomotion behaviors, both with and without objects. This dataset includes:

  • 8 different "props" (2 large balls, 2 small balls, 2 large boxes, 2 small boxes).
  • Interactions at 3 heights (floor, torso, head).
  • Each object/height combination involved picking up, carrying, and placing objects.
  • Additional data for walking and ball tossing.

The authors also mention using procedurally generated tasks for training and evaluation, which involve variations in object properties (size, mass), initial conditions, and task phases.

Key Insights and Ideas

  • Reusable Motor Primitives are Key: The core idea is that by distilling diverse motion capture into a general low-level motor skill module (NPMP), you create a powerful foundation that can be reused across many different tasks. This avoids retraining complex motor skills from scratch for every new task.
  • Egocentric Vision for Natural Interaction: Using first-person vision as input naturally leads to behaviors like gaze control that are crucial for interacting with the environment, without explicitly programming them.
  • The Power of Distillation: Distilling expert demonstrations into a single, compact module is more effective than trying to directly mimic individual demonstrations, especially for complex, multi-step tasks.
  • Task Variations as a Curriculum: Procedural generation of tasks with varying parameters (object properties, initial states) acts as an implicit curriculum, significantly improving the robustness and generalization of learned policies.
  • Balancing Specificity and Generality: There's a trade-off between using highly specific demonstrations for precise control and using more diverse data for broader generalization. The NPMP approach aims to strike a good balance.

Key Results

  • Successful Task Completion: The proposed approach successfully trained controllers for two challenging tasks: a "warehouse" task (instructed box carrying) and a "toss" task (ball catching and tossing into a bucket).
  • Vision vs. State Input: The system demonstrated effective performance using both egocentric vision and state features. Vision-based control showed an advantage in the warehouse task, while state-based control was slightly better for the toss task.
  • Robustness to Variations: The learned controllers showed robustness to procedural variations in object properties, initial positions, and task phases.
  • Comparison to Scratch Learning: Training from scratch for these tasks was significantly more challenging and required orders of magnitude more learning steps, often resulting in less complete or less robust behaviors (e.g., using a paddle-like motion instead of catching the ball).
  • Real-time Deployment: The resulting controllers can be deployed in real-time on a standard PC.

Limitations and Future Work

  • Data Efficiency: While better than training from scratch, the RL-based task policy training still requires a substantial amount of data.
  • Exploration Challenges: Even with the low-level skill module, exploration at the task policy level can be slow, especially with sparse rewards.
  • Curriculum Dependence: The warehouse task benefited significantly from specific initializations across different task phases, suggesting that carefully designed curricula are important.
  • Visual Quality vs. Performance: The authors note that slightly worse optimizers or termination conditions can lead to less "human-like" but higher-rewarding behaviors, indicating a trade-off between visual quality and task performance.
  • Sim-to-Real Transfer: The current work is limited to simulation. Future work could explore sim-to-real transfer to deploy these controllers on physical robots.
  • Architectural Choices: While the authors used simple MLPs for their encoder/decoder, exploring more advanced architectures for the NPMP module could potentially yield further improvements.
  • Hierarchical Skill Reuse: The idea of repeating the skill reuse concept hierarchically could lead to even more complex behaviors.

Summary updated

Abstract

We address the longstanding challenge of producing flexible, realistic humanoid character controllers that can perform diverse whole-body tasks involving object interactions. This challenge is central to a variety of fields, from graphics and animation to robotics and motor neuroscience. Our physics-based environment uses realistic actuation and first-person perception - including touch sensors and egocentric vision - with a view to producing active-sensing behaviors (e.g. gaze direction), transferability to real robots, and comparisons to the biology. We develop an integrated neural-network based approach consisting of a motor primitive module, human demonstrations, and an instructed reinforcement learning regime with curricula and task variations. We demonstrate the utility of our approach for several tasks, including goal-conditioned box carrying and ball catching, and we characterize its behavioral robustness. The resulting controllers can be deployed in real-time on a standard PC.1
BibTeX
@article{merel2020catch,
  author = {Josh Merel and Saran Tunyasuvunakool and Arun Ahuja and Yuval Tassa and Leonard Hasenclever and Vu Pham and Tom Erez and Greg Wayne and Nicolas Heess},
  title = {{Catch \& Carry: reusable neural controllers for vision-guided whole-body tasks}},
  journal = {ACM Trans. Graphics},
  year = {2020},
  doi = {10.1145/3386569.3392474},
}
Figure