Learning Reward Functions for Robotic Manipulation by Observing Humans

Minttu Alakuijala, Gabriel Dulac-Arnold, Julien Mairal, Jean Ponce, Cordelia Schmid

TL;DR

This paper introduces HOLD (Human Offline Learned Distances), a method that allows robots to learn complex manipulation tasks by watching unlabeled videos of humans. By learning a "functional distance" (how many time steps separate two states) from diverse human demonstrations, the authors create a dense reward signal that accelerates robot reinforcement learning by up to 18x and generalizes to entirely new robot embodiments.

What Problem Does This Paper Address?

Deep learning thrives on massive datasets, but robotics is stuck in a data bottleneck. Collecting millions of hours of robot interaction is prohibitively expensive and slow. While the internet is full of videos of humans performing tasks, using this data for robotics is notoriously difficult due to the domain gap:

  1. Visual Gap: A human arm looks nothing like a metallic Sawyer or Franka arm.
  2. Physical Gap: Human hands have different degrees of freedom and dynamics than robotic grippers.
  3. Missing Information: Videos don't come with "action labels" (joint torques or velocities) or explicit reward signals.

Most existing methods require either robot-specific data, manual task labels, or exact alignment between human and robot movements. The authors of this paper wanted to see if a robot could learn a useful reward function from "in-the-wild" human videos without any of these crutches.

Methods

The core idea is to learn a functional distance $d(s, g)$—an estimate of how many time steps an expert would take to get from current state $s$ to goal state $g$. The authors propose two ways to learn this from the Something-Something v2 dataset:

  • HOLD-R (Direct Regression): The model is trained to look at two frames from a video and directly predict the number of frames (time delta) between them.
  • HOLD-C (Time-Contrastive): Based on Time-Contrastive Networks (TCN), this method learns an embedding space where frames close in time are pulled together, and frames far apart are pushed away. Distance in this embedding space serves as the reward.

Once trained on human videos, the model is "frozen." When the robot tries a task, it is given a goal image of the completed task. The HOLD model compares the robot's current camera feed to that goal image. The closer the robot gets (in functional distance), the higher the reward it receives. This dense reward is then used to train a standard Reinforcement Learning (RL) agent (Soft Actor-Critic).

Datasets

  • Training: Something-Something v2 (SSv2). This is a massive crowd-sourced dataset containing over 220,000 video clips of humans performing 174 different action classes (e.g., "pushing something from left to right"). Crucially, the authors do not use the action labels during distance learning.
  • Evaluation: The models were tested on simulated robotic manipulation tasks from two benchmarks:
  • RLV: Pushing a puck and opening a drawer.
  • DVD: Closing a drawer, pushing a cup, and turning a faucet.

Key Insights and Ideas

  • Physics is the Universal Language: Even though humans and robots look different, the way objects move (displacements, rotations, collisions) follows the same laws of physics. By watching humans, the model learns the "flow" of a successful task.
  • Domain Randomization via Diversity: By training on thousands of different humans, backgrounds, and lighting conditions in SSv2, the model learns to ignore the "actor" and focus on the task progress. This allows it to generalize to a robot arm it has never seen before.
  • Temporal Regression as Reward: One of the most surprising findings is that simply predicting "how many seconds away is the goal?" (HOLD-R) provides a remarkably stable and effective reward signal for RL, often outperforming more complex contrastive methods.
  • Overcoming the "Puck Problem": The authors noted that distance models can sometimes focus too much on the robot arm's position rather than the object. They found that using the learned reward as an exploration prior alongside a sparse "success" reward is the most robust approach.

Key Results

The results demonstrate that watching humans provides a massive "head start" for robots:

  • Massive Acceleration: In the "Drawer Opening" task, HOLD-C reached success levels in 150k steps that took a standard sparse-reward agent over 2.75 million steps—a 18x speedup.
  • Zero-Shot Success: For some tasks, like "Close Drawer," the robot could learn to solve the task using only the human-learned reward, without any environment-specific success signal.
  • Superior Generalization: HOLD outperformed other state-of-the-art visual representations like R3M and TCN on these manipulation tasks, as shown in Figure 5.
Figure 5: HOLD-C outperforms TCN and R3M rewards on both RLV tasks.

As seen in Figure 3, the combination of HOLD rewards and sparse rewards (orange/purple lines) reaches high returns much faster than sparse rewards alone (blue line).

Figure 3: Return on the RLV tasks (5 random seeds, with standard error).

Limitations and Future Work

  • The "Copycat" Trap: The model can sometimes get distracted by the "fully actuated" parts of the scene (the arm). If the arm reaches the goal position but misses the object, the model might still give a high reward.
  • Long-Horizon Challenges: For very complex tasks, the distance metric becomes noisy. The authors had to break the "Drawer Opening" task into two sub-goals (reach handle, then pull) to get the best results.
  • Future Directions: The authors suggest that future work could involve better object-centric attention or using these distance models to automatically discover sub-goals for hierarchical RL.

This work moves us closer to a world where we can "program" robots simply by showing them a video of a human doing the job.

Summary updated

Abstract

Observing a human demonstrator manipulate objects provides a rich, scalable and inexpensive source of data for learning robotic policies. However, transferring skills from human videos to a robotic manipulator poses several challenges, not least a difference in action and observation spaces. In this work, we use unlabeled videos of humans solving a wide range of manipulation tasks to learn a task-agnostic reward function for robotic manipulation policies. Thanks to the diversity of this training data, the learned reward function sufficiently generalizes to image observations from a previously unseen robot embodiment and environment to provide a meaningful prior for directed exploration in reinforcement learning. We propose two methods for scoring states relative to a goal image: through direct temporal regression, and through distances in an embedding space obtained with time-contrastive learning. By conditioning the function on a goal image, we are able to reuse one model across a variety of tasks. Unlike prior work on leveraging human videos to teach robots, our method, Human Offline Learned Distances (HOLD) requires neither a priori data from the robot environment, nor a set of task-specific human demonstrations, nor a predefined notion of correspondence across morphologies, yet it is able to accelerate training of several manipulation tasks on a simulated robot arm compared to using only a sparse reward obtained from task completion.
BibTeX
@inproceedings{alakuijala2023learning,
  author = {Minttu Alakuijala and Gabriel Dulac-Arnold and Julien Mairal and Jean Ponce and Cordelia Schmid},
  title = {{Learning Reward Functions for Robotic Manipulation by Observing Humans}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  url = {https://openreview.net/forum?id=Z5Tn6ze4Jo},
}

Loading related content...

Figure