TL;DR
DoDont (Do’s and Don’ts) is a two-stage framework that supercharges unsupervised skill discovery by incorporating a tiny amount of human guidance—less than eight action-free videos. By training an "instruction network" to distinguish between desirable and undesirable behaviors, the algorithm steers agents toward complex, safe, and useful skills (like running upright) while avoiding "junk" behaviors (like tripping or rolling) that plague purely unsupervised methods.
What Problem Does This Paper Address?
Unsupervised Skill Discovery (USD) aims to create "foundational policies" that can perform a wide variety of tasks without needing a specific reward function for each one. However, USD faces two major hurdles in complex environments:
- Complexity Gap: In high-dimensional spaces (like a multi-jointed quadruped), agents often settle for simple behaviors like standing or wiggling because complex movements like running are harder to discover by chance.
- Safety and Utility: Purely unsupervised agents don't know the difference between a "good" diverse behavior and a "bad" one. They might learn to flip over or fall into pits simply because those states are novel, which is useless or even damaging in the real world.
Hand-designing reward functions to fix this is notoriously difficult to scale. The authors ask: Can we use a handful of raw videos to tell the agent what we want, without losing the benefits of unsupervised diversity?
Methods
DoDont operates in two distinct stages, building upon Distance-maximizing Skill Discovery (DSD):
- Instruction Learning: The researchers collect a small set of "Do" videos (desirable behavior) and "Don't" videos (undesirable behavior). They train a binary classifier—the instruction network ($\hat{p}_\psi$)—to look at pairs of adjacent states and predict whether the transition belongs to a "Do" or a "Don't" video.
- Skill Learning: This network is then integrated into a DSD algorithm (specifically METRA). In standard DSD, the agent tries to maximize the distance between states to ensure diversity. DoDont uses the instruction network to re-weight the reward.
Crucially, the authors found that multiplying the intrinsic diversity reward by the instruction network's output (Equation 7 in the paper) works significantly better than simply adding them. This ensures that the agent only receives a "diversity bonus" when it is performing behaviors that look like the "Do" instructions.
Datasets
The authors evaluated DoDont across several challenging continuous control benchmarks:
- DeepMind Control (DMC) Suite: Specifically the Cheetah and Quadruped environments, which require complex coordination.
- Kitchen Environment: A manipulation task involving six sub-tasks (e.g., opening a microwave, turning on a burner).
- ExORL Datasets: Used for offline zero-shot RL experiments to show that DoDont can also prioritize behaviors within pre-collected, unlabeled datasets.
- Instruction Data: The "Do" videos were generated using expert policies, while "Don't" videos consisted of random actions or specific failures (like flipping). Most experiments used only 8 videos total.
Key Insights and Ideas
- Action-Free Guidance: Unlike imitation learning, DoDont doesn't need to know which actions the expert took. It only needs to see the resulting states (pixels), making it much easier to collect data.
- The Power of "Don'ts": The paper highlights that penalizing bad behaviors is just as important as encouraging good ones. Without "Don't" labels, agents often fail to avoid risky or undesirable actions that happen to be high-entropy.
- Diversity + Intention: Even if the "Do" videos only show an agent running in one direction, the underlying USD objective pushes the agent to discover how to run in all directions. The instruction network acts as a filter that says "running is good," and the USD objective says "now find every way to run."
- Multiplicative Scaling: By using the instruction network as a coefficient for the intrinsic reward, the system automatically balances the two objectives without the heavy hyperparameter tuning required by additive methods.
Key Results
- Complex Locomotion: In the Quadruped environment, standard METRA agents often learned to roll on their backs to move. DoDont agents learned to run upright, achieving significantly higher state coverage and task rewards.
- Hazard Avoidance: When the left side of an environment was designated as "hazardous" via "Don't" videos, DoDont agents successfully learned to explore only the safe right side, whereas baseline USD agents explored both equally.
- Manipulation: In the Kitchen environment, DoDont successfully learned to perform more tasks (higher policy coverage) than baselines like SMERL or DGPO.
- Data Efficiency: Ablation studies showed the method remains effective even with as few as two videos (one "Do" and one "Don't").
Limitations and Future Work
- In-Domain Requirement: Currently, the instruction videos must be recorded in the same environment the agent is training in. The authors suggest that using "in-the-wild" videos (e.g., from YouTube) or videos generated by models like Sora is a promising next step.
- Simulation Dependency: While the supervision is minimal, the agent still requires a large number of environment steps in simulation to learn the skills.
- State vs. Pixels: While the instruction network uses pixels, the policy training in some experiments still relied on state information. Moving to a fully pixel-to-action pipeline for all tasks remains a goal for future scalability.
Summary updated