Do's and Don'ts: Learning Desirable Skills with Instruction Videos

Hyunseung Kim, Byungkun Lee, Hojoon Lee, Dongyoon Hwang, Donghu Kim, Jaegul Choo

TL;DR

DoDont (Do’s and Don’ts) is a two-stage framework that supercharges unsupervised skill discovery by incorporating a tiny amount of human guidance—less than eight action-free videos. By training an "instruction network" to distinguish between desirable and undesirable behaviors, the algorithm steers agents toward complex, safe, and useful skills (like running upright) while avoiding "junk" behaviors (like tripping or rolling) that plague purely unsupervised methods.

What Problem Does This Paper Address?

Unsupervised Skill Discovery (USD) aims to create "foundational policies" that can perform a wide variety of tasks without needing a specific reward function for each one. However, USD faces two major hurdles in complex environments:

  1. Complexity Gap: In high-dimensional spaces (like a multi-jointed quadruped), agents often settle for simple behaviors like standing or wiggling because complex movements like running are harder to discover by chance.
  2. Safety and Utility: Purely unsupervised agents don't know the difference between a "good" diverse behavior and a "bad" one. They might learn to flip over or fall into pits simply because those states are novel, which is useless or even damaging in the real world.

Hand-designing reward functions to fix this is notoriously difficult to scale. The authors ask: Can we use a handful of raw videos to tell the agent what we want, without losing the benefits of unsupervised diversity?

Methods

DoDont operates in two distinct stages, building upon Distance-maximizing Skill Discovery (DSD):

  1. Instruction Learning: The researchers collect a small set of "Do" videos (desirable behavior) and "Don't" videos (undesirable behavior). They train a binary classifier—the instruction network ($\hat{p}_\psi$)—to look at pairs of adjacent states and predict whether the transition belongs to a "Do" or a "Don't" video.
  2. Skill Learning: This network is then integrated into a DSD algorithm (specifically METRA). In standard DSD, the agent tries to maximize the distance between states to ensure diversity. DoDont uses the instruction network to re-weight the reward.
Figure 1: (a) The offline instruction video dataset includes videos of desirable behaviors (Do’s) and undesirable behaviors (Don’ts). (b) Unsupervised skill discovery algorithms tend to learn undesirable behaviors. (c) In DoDont, an instruction network is first trained with the Do’s and Don’ts videos to distinguish desirable and undesirable behaviors. Then, this instruction network adjusts the intrinsic reward of the skill discovery algorithm, promoting desirable skills while avoiding undesirable ones.

Crucially, the authors found that multiplying the intrinsic diversity reward by the instruction network's output (Equation 7 in the paper) works significantly better than simply adding them. This ensures that the agent only receives a "diversity bonus" when it is performing behaviors that look like the "Do" instructions.

Datasets

The authors evaluated DoDont across several challenging continuous control benchmarks:

  • DeepMind Control (DMC) Suite: Specifically the Cheetah and Quadruped environments, which require complex coordination.
  • Kitchen Environment: A manipulation task involving six sub-tasks (e.g., opening a microwave, turning on a burner).
  • ExORL Datasets: Used for offline zero-shot RL experiments to show that DoDont can also prioritize behaviors within pre-collected, unlabeled datasets.
  • Instruction Data: The "Do" videos were generated using expert policies, while "Don't" videos consisted of random actions or specific failures (like flipping). Most experiments used only 8 videos total.

Key Insights and Ideas

  • Action-Free Guidance: Unlike imitation learning, DoDont doesn't need to know which actions the expert took. It only needs to see the resulting states (pixels), making it much easier to collect data.
  • The Power of "Don'ts": The paper highlights that penalizing bad behaviors is just as important as encouraging good ones. Without "Don't" labels, agents often fail to avoid risky or undesirable actions that happen to be high-entropy.
  • Diversity + Intention: Even if the "Do" videos only show an agent running in one direction, the underlying USD objective pushes the agent to discover how to run in all directions. The instruction network acts as a filter that says "running is good," and the USD objective says "now find every way to run."
  • Multiplicative Scaling: By using the instruction network as a coefficient for the intrinsic reward, the system automatically balances the two objectives without the heavy hyperparameter tuning required by additive methods.

Key Results

  • Complex Locomotion: In the Quadruped environment, standard METRA agents often learned to roll on their backs to move. DoDont agents learned to run upright, achieving significantly higher state coverage and task rewards.
  • Hazard Avoidance: When the left side of an environment was designated as "hazardous" via "Don't" videos, DoDont agents successfully learned to explore only the safe right side, whereas baseline USD agents explored both equally.
  • Manipulation: In the Kitchen environment, DoDont successfully learned to perform more tasks (higher policy coverage) than baselines like SMERL or DGPO.
  • Data Efficiency: Ablation studies showed the method remains effective even with as few as two videos (one "Do" and one "Don't").

Limitations and Future Work

  • In-Domain Requirement: Currently, the instruction videos must be recorded in the same environment the agent is training in. The authors suggest that using "in-the-wild" videos (e.g., from YouTube) or videos generated by models like Sora is a promising next step.
  • Simulation Dependency: While the supervision is minimal, the agent still requires a large number of environment steps in simulation to learn the skills.
  • State vs. Pixels: While the instruction network uses pixels, the policy training in some experiments still relied on state information. Moving to a fully pixel-to-action pipeline for all tasks remains a goal for future scalability.

Summary updated

Abstract

Unsupervised skill discovery is a learning paradigm that aims to acquire diverse behaviors without explicit rewards. However, it faces challenges in learning complex behaviors and often leads to learning unsafe or undesirable behaviors. For instance, in various continuous control tasks, current unsupervised skill discovery methods succeed in learning basic locomotions like standing but struggle with learning more complex movements such as walking and running. Moreover, they may acquire unsafe behaviors like tripping and rolling or navigate to undesirable locations such as pitfalls or hazardous areas. In response, we present DoDont (Do's and Don'ts), an instruction-based skill discovery algorithm composed of two stages. First, in an instruction learning stage, DoDont leverages action-free instruction videos to train an instruction network to distinguish desirable transitions from undesirable ones. Then, in the skill learning stage, the instruction network adjusts the reward function of the skill discovery algorithm to weight the desired behaviors. Specifically, we integrate the instruction network into a distance-maximizing skill discovery algorithm, where the instruction network serves as the distance function. Empirically, with less than 8 instruction videos, DoDont effectively learns desirable behaviors and avoids undesirable ones across complex continuous control tasks. Code and videos are available at https://mynsng.github.io/dodont/
BibTeX
@inproceedings{kim2024do,
  author = {Hyunseung Kim and Byungkun Lee and Hojoon Lee and Dongyoon Hwang and Donghu Kim and Jaegul Choo},
  title = {{Do's and Don'ts: Learning Desirable Skills with Instruction Videos}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=7X5zu6GIuW},
}
Figure