Learning from Limited Demonstrations

Beomjoon Kim, Amir-massoud Farahmand, Joelle Pineau, Doina PrecupNeurIPS 2013 · OpenReview

Introduction

Learning complex behaviors for robots typically relies on either Learning from Demonstration (LfD), where an agent mimics an expert, or Reinforcement Learning (RL), where an agent learns through trial and error. Both approaches face significant real-world bottlenecks. LfD algorithms often assume the expert is optimal and demonstrations are abundant, which is rarely true since collecting data is expensive and experts make mistakes. Conversely, pure RL requires massive amounts of interaction data to converge and can be reckless during exploration, risking catastrophic failures in physical systems.

To bridge this gap, the authors introduce Approximate Policy Iteration with Demonstration (APID). APID is a hybrid algorithm that treats expert demonstrations as soft constraints within a regularized policy iteration framework. It asks a fundamental question: How can we use a small amount of potentially imperfect expert advice to safely and efficiently guide an RL agent toward a good policy?

Methods: Approximate Policy Iteration with Demonstration

The core of the paper is APID, which builds on Approximate Policy Iteration (API)—a standard RL technique that alternates between evaluating a policy (finding the value of actions) and improving it.

The key innovation lies in the policy evaluation step. Instead of solely minimizing the Bellman error (how well the value function predicts future rewards), APID incorporates large-margin constraints derived from expert data:

  • The Constraint: For any state with an expert demonstration, the algorithm ensures the expert's action has a higher $Q$-value than any other action by at least a fixed margin.
  • Slack Variables: Similar to Support Vector Machines, APID uses "slack variables." If the expert's action is actually suboptimal, the algorithm pays a penalty to ignore the constraint, allowing actual environmental rewards to override poor expert advice.
  • Regularization: A regularized objective function prevents overfitting, ensuring the learned policy generalizes to unvisited states.

Rather than strictly mimicking the expert, APID uses demonstrations to "shape" the value function. The expert provides a rough map of high-value areas, and the RL agent fills in the details through interaction.

Experimental Setup

The authors evaluated APID on two main benchmarks:

  1. Car Brake Control Simulation: A continuous-state simulation where an agent must reach and maintain a target velocity. This tested robustness against varying quantities and qualities of expert data.
  2. Real Robot Path Finding: An iRobot Create equipped with a Kinect sensor navigating a cluttered room. This served as a live system test where exploration data is expensive and demonstrations are limited to a single trajectory.

Key Results

APID demonstrated significant improvements in sample efficiency, stability, and robustness compared to baseline methods.

Performance comparison of APID against LSPI and Supervised Learning
  • Lower Variance and Faster Convergence: Compared to pure RL (LSPI), APID showed much lower variance in rewards across training iterations. The expert constraints act as a stabilizer. In the robot task, APID reached the goal after only a few iterations of exploration, whereas pure LSPI wasted time exploring irrelevant areas of the state space.
  • Real-World Success: In the robot navigation task, APID reached the goal in 36.1 ± 0.24 seconds, nearly identical to the human expert's time of 35.9 seconds. Pure supervised learning and pure LSPI failed to consistently reach the goal within the time limit.
Performance comparison of APID against LSPI and DAgger
  • Outperforming DAgger with Suboptimal Experts: Because of the slack variables, APID is significantly more robust than state-of-the-art LfD algorithms like DAgger. In scenarios where experts provided random or suboptimal actions (e.g., 25% of the time), DAgger's performance plummeted. APID, however, continued to improve by leveraging RL signals to eventually ignore the bad advice.

Limitations and Future Work

While highly effective, APID has a few limitations. The hyperparameters $\alpha$ (balancing RL vs. expert data) and the regularization coefficient $\lambda$ currently require manual tuning; automating this via model selection is a necessary next step. Theoretically, the provided upper bound on Bellman error is less informative when the number of expert demonstrations far exceeds the number of RL samples. Finally, for safety-critical applications, the authors suggest exploring varying margins ($\Delta_i$) to enforce stricter adherence to expert behavior in high-risk states, such as near a cliff.

Summary updated

Abstract

We propose an approach to learning from demonstration (LfD) which leverages expert data, even if the expert examples are very few or inaccurate. We achieve this by integrating LfD in an approximate policy iteration algorithm. The key idea of our approach is that expert examples are used to generate linear constraints on the optimization, in a similar fashion to large-margin classification. We prove an upper bound on the true Bellman error of the approximation computed by the algorithm at each iteration. We show empirically that the algorithm outperforms both pure policy iteration, as well as DAgger (a state-of-art LfD algorithm) and supervised learning in a variety of scenarios, including when very few and/or imperfect demonstrations are available. Our experiments include simulations as well as a real robotic navigation task.
BibTeX
@inproceedings{kim2013learning,
  author = {Beomjoon Kim and Amir-massoud Farahmand and Joelle Pineau and Doina Precup},
  title = {{Learning from Limited Demonstrations}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2013},
  url = {https://openreview.net/forum?id=BJEq-K-_-r},
}
Figure