Better-than-Demonstrator Imitation Learning via Automatically-Ranked Demonstrations

Daniel S. Brown, Wonjoon Goo, Scott NiekumCoRL 2019 · OpenReview

Abstract

The performance of imitation learning is typically upper-bounded by the performance of the demonstrator. While recent empirical results demonstrate that ranked demonstrations allow for better-than-demonstrator performance, obtaining these rankings usually requires significant human effort. We introduce D-REX (Disturbance-based Reward Extrapolation), a method that automatically generates a ranked dataset of trajectories by injecting noise into a cloned policy. This allows an agent to learn a reward function that extrapolates beyond the demonstrator's skill level, achieving state-of-the-art performance on several benchmarks without human supervision or ground-truth rewards.

Introduction

In traditional Imitation Learning (IL), the goal is to mimic a demonstrator. However, this creates a "performance ceiling": the agent is typically only as good as its teacher. If the demonstrations are suboptimal—due to human error, lack of precision, or teleoperation lag—the agent will inherit those flaws. While some recent methods (like T-REX) can exceed demonstrator performance by using ranked demonstrations, these rankings usually require tedious human supervision. D-REX fills this gap by providing an algorithm to automate the ranking process through noise injection.

Methodology

The core of the paper is D-REX, which operates in three main stages:

1. Behavioral Cloning (BC)

The agent first learns a simple policy ($\pi_{BC}$) that mimics the provided unlabeled and potentially suboptimal demonstrations.

2. Automatic Ranking via Noise Injection

The authors hypothesize that if $\pi_{BC}$ is better than random, then adding increasing amounts of noise to it will result in increasingly worse performance. They generate a suite of trajectories with varying levels of $\epsilon$-greedy noise. Because the noise levels are known, the system can automatically rank these trajectories from "best" (no noise) to "worst" (high noise).

Automatic ranking generation through noise injection

3. Reward Extrapolation and Optimization

Using these automatically generated rankings, a neural network reward function is trained via the T-REX framework. The network learns to assign higher values to features prevalent in low-noise trajectories and lower values to those in high-noise ones. Finally, a standard Reinforcement Learning algorithm (PPO) is used to optimize a new policy against this learned reward function.

Reward function learned from ranked trajectories

Key Insights

  • Rankings Reduce Ambiguity: A total ranking over policies reduces reward function ambiguity more effectively than standard Inverse Reinforcement Learning (IRL). While IRL tries to find a reward that makes the teacher look "optimal," rankings provide a gradient that shows the direction of improvement.
  • Noise as a Proxy for Suboptimality: By intentionally breaking a cloned policy, the system learns what "bad" looks like, which implicitly defines what "better" looks like.
  • Feature Selection: Noise injection acts as a natural feature selector. If a feature disappears as noise increases, the learner identifies it as potentially high-value.

Experimental Results

D-REX achieved better-than-demonstrator performance in 8 out of 9 tasks across MuJoCo and Atari 2600 benchmarks.

  • Significant Gains: In the HalfCheetah environment, D-REX achieved a 418% increase in performance over the best demonstration.
  • Qualitative Success: In Seaquest, the learned reward function successfully identified that avoiding enemies is a high-reward behavior, even though the demonstrator frequently collided with them.
  • Discovery of Loopholes: In Qbert*, D-REX discovered a loophole in the game logic that allowed it to rack up nearly infinite points, far exceeding any human or algorithmic demonstration provided.

Conclusion and Future Work

D-REX demonstrates that agents can learn to outperform their teachers by synthesizing their own failure cases. While the method relies on the monotonicity assumption (that noise decreases performance), it provides a powerful, unsupervised way to break the performance ceiling of imitation learning. Future work will explore applying D-REX to "Imitation from Observation" settings where only video data is available.

Summary updated

Abstract

The performance of imitation learning is typically upper-bounded by the performance of the demonstrator. While recent empirical results demonstrate that ranked demonstrations allow for better-than-...
BibTeX
@inproceedings{brown2019better,
  author = {Daniel S. Brown and Wonjoon Goo and Scott Niekum},
  title = {{Better-than-Demonstrator Imitation Learning via Automatically-Ranked Demonstrations}},
  booktitle = {Conference on Robot Learning},
  year = {2019},
  url = {https://openreview.net/forum?id=-U8IQwk28aQ},
}
Figure