TL;DR
This paper presents a framework that allows robots to learn complex, multi-step tasks from just a single demonstration. By "localizing" additional examples of task sub-steps within unlabelled auxiliary videos (like YouTube clips), the system autonomously gathers the data it needs to infer reward functions and train robust robotic policies without requiring massive, hand-labeled datasets.
What Problem Does This Paper Address?
Learning from Demonstration (LfD) is the "holy grail" of intuitive robot programming, but it faces a massive bottleneck: data requirements. Humans can watch a single video of someone making coffee and understand the steps involved because we have a lifetime of context. Robots, however, usually require hundreds of demonstrations in varying conditions to generalize effectively.
Furthermore, most current "Learning from Observation" (LfO) methods—where the robot learns from video pixels without access to the demonstrator's internal motor commands—only handle simple, single-step tasks. Multi-step tasks (like cooking or assembly) are much harder because each step might have different goals and involve different objects. If a robot tries to learn one giant policy for a complex task from a single demo, it almost inevitably overfits and fails in new situations.
Methods
The authors propose a four-stage pipeline to bridge this "context gap":
- User-Segmented Demonstration: A human provides one video demonstration of a multi-step task, manually marking where one sub-task (e.g., "pick up cup") ends and the next (e.g., "pour water") begins.
- One-Shot Activity Localization: This is the "secret sauce." Using meta-learning techniques (MAML and Reptile), the system takes those single snippets from the user's demo and searches through "auxiliary" (unsegmented, unlabelled) videos to find more examples of those specific actions.
- Reward Function Inference: Once it has a collection of snippets for a sub-task, the system learns what "progress" looks like. It uses a shuffle-and-learn approach, training a neural network to predict the temporal order of frames. If the network can tell that frame B comes after frame A, it has essentially learned a surrogate reward function: "moving toward the goal is good."
- Reinforcement Learning (RL): Finally, the robot uses Proximal Policy Optimization (PPO) to learn a control policy for each sub-task based on these inferred rewards. These modular policies are then chained together to complete the full task.
Datasets
The researchers evaluated their system on two distinct fronts:
- Reacher Environment: A simulated two-joint robotic arm task. The robot had to reach colored targets in specific sequences. This provided a controlled environment to test the full pipeline from pixels to policy.
- ActivityNet: A large-scale dataset of 100 different human activities (e.g., playing golf, drinking coffee). This was used to prove that their one-shot localization algorithm could handle the noise and high variance of real-world, unconstrained video.
Key Insights and Ideas
- Modularization is Key: Instead of learning one complex policy, the authors argue for learning separate policies for each "primitive" action. This makes the learning problem more tractable and improves generalization.
- Auxiliary Video as Context: The paper demonstrates that robots don't need perfectly curated data. They can leverage "web-scale" video data to understand the underlying physics and progression of a task, even if the video doesn't perfectly match their own environment.
- Simpler is Often Better: For reward inference, the authors found that using simple ordered pairs of frames outperformed more complex "triplet losses" used in prior work (like Time-Contrastive Networks), while also requiring fewer hyperparameters.
- Robustness to "Label Noise": Remarkably, the reward inference was robust enough to work even when the activity localization wasn't perfect. As long as the majority of the found snippets were relevant, the robot could still figure out the task.
Key Results
- Localization Accuracy: The meta-learning approach (MAML) significantly outperformed standard classifiers. On the Reacher task, it achieved a mean Intersection-over-Union (mIoU) of 0.84 compared to just 0.48 for the baseline.
- Policy Success: In the multi-step reaching task, the robot achieved a success rate of roughly 85% (for the orange target) when using localized auxiliary video.
- The "One-Shot" Failure: Without the auxiliary video (learning from only the single demo), the success rate plummeted to a measly 4%. This highlights that a single demonstration simply doesn't contain enough information for a robot to generalize without external context.
Limitations and Future Work
While promising, the authors note several hurdles for moving this into the real world:
- Visual Variance: Real-world videos have massive shifts in camera angles (e.g., third-person vs. egocentric) and lighting that the current system might struggle with.
- Temporal Context: The current "snippet-based" classification doesn't look at the broader timeline of a video, which could lead to "flickering" or noisy predictions in complex scenes.
- Multi-Modal Learning: Future iterations could benefit from incorporating other data types found in web videos, such as audio or text descriptions/subtitles, to better identify and segment tasks.
Summary updated