One-Shot Learning of Multi-Step Tasks from Observation via Activity Localization in Auxiliary Video

Wonjoon Goo, Scott Niekum

TL;DR

This paper presents a framework that allows robots to learn complex, multi-step tasks from just a single demonstration. By "localizing" additional examples of task sub-steps within unlabelled auxiliary videos (like YouTube clips), the system autonomously gathers the data it needs to infer reward functions and train robust robotic policies without requiring massive, hand-labeled datasets.

What Problem Does This Paper Address?

Learning from Demonstration (LfD) is the "holy grail" of intuitive robot programming, but it faces a massive bottleneck: data requirements. Humans can watch a single video of someone making coffee and understand the steps involved because we have a lifetime of context. Robots, however, usually require hundreds of demonstrations in varying conditions to generalize effectively.

Furthermore, most current "Learning from Observation" (LfO) methods—where the robot learns from video pixels without access to the demonstrator's internal motor commands—only handle simple, single-step tasks. Multi-step tasks (like cooking or assembly) are much harder because each step might have different goals and involve different objects. If a robot tries to learn one giant policy for a complex task from a single demo, it almost inevitably overfits and fails in new situations.

Methods

The authors propose a four-stage pipeline to bridge this "context gap":

  1. User-Segmented Demonstration: A human provides one video demonstration of a multi-step task, manually marking where one sub-task (e.g., "pick up cup") ends and the next (e.g., "pour water") begins.
  2. One-Shot Activity Localization: This is the "secret sauce." Using meta-learning techniques (MAML and Reptile), the system takes those single snippets from the user's demo and searches through "auxiliary" (unsegmented, unlabelled) videos to find more examples of those specific actions.
  3. Reward Function Inference: Once it has a collection of snippets for a sub-task, the system learns what "progress" looks like. It uses a shuffle-and-learn approach, training a neural network to predict the temporal order of frames. If the network can tell that frame B comes after frame A, it has essentially learned a surrogate reward function: "moving toward the goal is good."
  4. Reinforcement Learning (RL): Finally, the robot uses Proximal Policy Optimization (PPO) to learn a control policy for each sub-task based on these inferred rewards. These modular policies are then chained together to complete the full task.

Datasets

The researchers evaluated their system on two distinct fronts:

  • Reacher Environment: A simulated two-joint robotic arm task. The robot had to reach colored targets in specific sequences. This provided a controlled environment to test the full pipeline from pixels to policy.
  • ActivityNet: A large-scale dataset of 100 different human activities (e.g., playing golf, drinking coffee). This was used to prove that their one-shot localization algorithm could handle the noise and high variance of real-world, unconstrained video.

Key Insights and Ideas

  • Modularization is Key: Instead of learning one complex policy, the authors argue for learning separate policies for each "primitive" action. This makes the learning problem more tractable and improves generalization.
  • Auxiliary Video as Context: The paper demonstrates that robots don't need perfectly curated data. They can leverage "web-scale" video data to understand the underlying physics and progression of a task, even if the video doesn't perfectly match their own environment.
  • Simpler is Often Better: For reward inference, the authors found that using simple ordered pairs of frames outperformed more complex "triplet losses" used in prior work (like Time-Contrastive Networks), while also requiring fewer hyperparameters.
  • Robustness to "Label Noise": Remarkably, the reward inference was robust enough to work even when the activity localization wasn't perfect. As long as the majority of the found snippets were relevant, the robot could still figure out the task.

Key Results

  • Localization Accuracy: The meta-learning approach (MAML) significantly outperformed standard classifiers. On the Reacher task, it achieved a mean Intersection-over-Union (mIoU) of 0.84 compared to just 0.48 for the baseline.
  • Policy Success: In the multi-step reaching task, the robot achieved a success rate of roughly 85% (for the orange target) when using localized auxiliary video.
  • The "One-Shot" Failure: Without the auxiliary video (learning from only the single demo), the success rate plummeted to a measly 4%. This highlights that a single demonstration simply doesn't contain enough information for a robot to generalize without external context.

Limitations and Future Work

While promising, the authors note several hurdles for moving this into the real world:

  • Visual Variance: Real-world videos have massive shifts in camera angles (e.g., third-person vs. egocentric) and lighting that the current system might struggle with.
  • Temporal Context: The current "snippet-based" classification doesn't look at the broader timeline of a video, which could lead to "flickering" or noisy predictions in complex scenes.
  • Multi-Modal Learning: Future iterations could benefit from incorporating other data types found in web videos, such as audio or text descriptions/subtitles, to better identify and segment tasks.

Summary updated

Abstract

Due to burdensome data requirements, learning from demonstration often falls short of its promise to allow users to quickly and naturally program robots. Demonstrations are inherently ambiguous and incomplete, making correct generalization to unseen situations difficult without a large number of demonstrations in varying conditions. By contrast, humans are often able to learn complex tasks from a single demonstration (typically observations without action labels) by leveraging context learned over a lifetime. Inspired by this capability, our goal is to enable robots to perform one-shot learning of multi-step tasks from observation by leveraging auxiliary video data as context. Our primary contribution is a novel system that achieves this goal by: (1) using a single user-segmented demonstration to define the primitive actions that comprise a task, (2) localizing additional examples of these actions in unsegmented auxiliary videos via a metalearning-based approach, (3) using these additional examples to learn a reward function for each action, and (4) performing reinforcement learning on top of the inferred reward functions to learn action policies that can be combined to accomplish the task. We empirically demonstrate that a robot can learn multi-step tasks more effectively when provided auxiliary video, and that performance greatly improves when localizing individual actions, compared to learning from unsegmented videos.
BibTeX
@inproceedings{goo2019one,
  author = {Wonjoon Goo and Scott Niekum},
  title = {{One-Shot Learning of Multi-Step Tasks from Observation via Activity Localization in Auxiliary Video}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2019},
  url = {https://openreview.net/forum?id=yqx29-DAw4U},
}

Loading related content...

Figure