TL;DR
This paper introduces HOLD (Human Offline Learned Distances), a method that allows robots to learn complex manipulation tasks by watching unlabeled videos of humans. By learning a "functional distance" (how many time steps separate two states) from diverse human demonstrations, the authors create a dense reward signal that accelerates robot reinforcement learning by up to 18x and generalizes to entirely new robot embodiments.
What Problem Does This Paper Address?
Deep learning thrives on massive datasets, but robotics is stuck in a data bottleneck. Collecting millions of hours of robot interaction is prohibitively expensive and slow. While the internet is full of videos of humans performing tasks, using this data for robotics is notoriously difficult due to the domain gap:
- Visual Gap: A human arm looks nothing like a metallic Sawyer or Franka arm.
- Physical Gap: Human hands have different degrees of freedom and dynamics than robotic grippers.
- Missing Information: Videos don't come with "action labels" (joint torques or velocities) or explicit reward signals.
Most existing methods require either robot-specific data, manual task labels, or exact alignment between human and robot movements. The authors of this paper wanted to see if a robot could learn a useful reward function from "in-the-wild" human videos without any of these crutches.
Methods
The core idea is to learn a functional distance $d(s, g)$—an estimate of how many time steps an expert would take to get from current state $s$ to goal state $g$. The authors propose two ways to learn this from the Something-Something v2 dataset:
- HOLD-R (Direct Regression): The model is trained to look at two frames from a video and directly predict the number of frames (time delta) between them.
- HOLD-C (Time-Contrastive): Based on Time-Contrastive Networks (TCN), this method learns an embedding space where frames close in time are pulled together, and frames far apart are pushed away. Distance in this embedding space serves as the reward.
Once trained on human videos, the model is "frozen." When the robot tries a task, it is given a goal image of the completed task. The HOLD model compares the robot's current camera feed to that goal image. The closer the robot gets (in functional distance), the higher the reward it receives. This dense reward is then used to train a standard Reinforcement Learning (RL) agent (Soft Actor-Critic).
Datasets
- Training: Something-Something v2 (SSv2). This is a massive crowd-sourced dataset containing over 220,000 video clips of humans performing 174 different action classes (e.g., "pushing something from left to right"). Crucially, the authors do not use the action labels during distance learning.
- Evaluation: The models were tested on simulated robotic manipulation tasks from two benchmarks:
- RLV: Pushing a puck and opening a drawer.
- DVD: Closing a drawer, pushing a cup, and turning a faucet.
Key Insights and Ideas
- Physics is the Universal Language: Even though humans and robots look different, the way objects move (displacements, rotations, collisions) follows the same laws of physics. By watching humans, the model learns the "flow" of a successful task.
- Domain Randomization via Diversity: By training on thousands of different humans, backgrounds, and lighting conditions in SSv2, the model learns to ignore the "actor" and focus on the task progress. This allows it to generalize to a robot arm it has never seen before.
- Temporal Regression as Reward: One of the most surprising findings is that simply predicting "how many seconds away is the goal?" (HOLD-R) provides a remarkably stable and effective reward signal for RL, often outperforming more complex contrastive methods.
- Overcoming the "Puck Problem": The authors noted that distance models can sometimes focus too much on the robot arm's position rather than the object. They found that using the learned reward as an exploration prior alongside a sparse "success" reward is the most robust approach.
Key Results
The results demonstrate that watching humans provides a massive "head start" for robots:
- Massive Acceleration: In the "Drawer Opening" task, HOLD-C reached success levels in 150k steps that took a standard sparse-reward agent over 2.75 million steps—a 18x speedup.
- Zero-Shot Success: For some tasks, like "Close Drawer," the robot could learn to solve the task using only the human-learned reward, without any environment-specific success signal.
- Superior Generalization: HOLD outperformed other state-of-the-art visual representations like R3M and TCN on these manipulation tasks, as shown in Figure 5.
As seen in Figure 3, the combination of HOLD rewards and sparse rewards (orange/purple lines) reaches high returns much faster than sparse rewards alone (blue line).
Limitations and Future Work
- The "Copycat" Trap: The model can sometimes get distracted by the "fully actuated" parts of the scene (the arm). If the arm reaches the goal position but misses the object, the model might still give a high reward.
- Long-Horizon Challenges: For very complex tasks, the distance metric becomes noisy. The authors had to break the "Drawer Opening" task into two sub-goals (reach handle, then pull) to get the best results.
- Future Directions: The authors suggest that future work could involve better object-centric attention or using these distance models to automatically discover sub-goals for hierarchical RL.
This work moves us closer to a world where we can "program" robots simply by showing them a video of a human doing the job.
Summary updated