Interactive Reinforcement Learning with Dynamic Reuse of Prior Knowledge from Human and Agent Demonstrations

Zhaodong Wang, Matthew E. TaylorIJCAI 2019 · OpenReview

TL;DR

This paper introduces DRoP, a novel algorithm for reinforcement learning that dynamically balances reusing prior knowledge from demonstrations with learning a new policy. DRoP uses an online confidence measure to adaptively decide how much to trust demonstrations, leading to faster learning and improved performance, and can even intelligently request new demonstrations when needed.

What Problem Does This Paper Address?

Reinforcement learning (RL) agents often require vast amounts of data to achieve good performance, which can be impractical in real-world scenarios. While demonstrations from humans or other agents can significantly speed up learning, simply mimicking these demonstrations can limit an agent's performance, especially if the demonstrations are suboptimal. Existing methods for leveraging demonstrations often struggle to adapt to varying demonstration quality, handle inconsistencies, or balance prior knowledge with self-learning effectively. This paper tackles the challenge of efficiently and adaptively using prior knowledge from demonstrations to accelerate RL while allowing the agent to eventually surpass the demonstrator.

Methods

The core of this paper is the Dynamic Reuse of Prior (DRoP) algorithm. DRoP operates in three main steps:

  1. Demonstration Collection: Collects state-action pairs from human or agent demonstrations.
  2. Policy Training: Trains an initial policy model (e.g., Gaussian regression or a neural network) on this demonstration data.
  3. Dynamic Reuse and Confidence Update: This is the novel part. DRoP uses a temporal difference (TD) approach to maintain online confidence measures for both the prior knowledge (from demonstrations) and the agent's own learned policy (e.g., Q-learning).
  4. Confidence Models: Two confidence models are maintained: $CP(s)$ for prior knowledge and $CQ(s)$ for the agent's learned policy. These are updated using TD learning based on the agent's experience.
  5. Action Selection: A "soft-hard-∈" (S-H-∈) decision model is used to select actions. This model dynamically balances between greedily exploiting the higher confidence source (hard decision) and probabilistically exploring based on confidence values (soft decision). This allows the agent to adapt to situations where the prior might be misleading or the learned policy is still uncertain.
  6. Multiple Demonstrators: DRoP can handle multiple sources of prior knowledge by expanding the action selection mechanism to weigh confidence from each source.

A key innovation is the online confidence measurement, which adapts to the agent's experience, unlike previous methods that relied on offline confidence or fixed decay rates.

Datasets

The paper evaluates DRoP on two standard domains:

  • Cartpole: A classic control problem from OpenAI Gym.
  • Mario: A more complex domain based on Nintendo's Mario Brothers, using a randomly sampled world generator for diversity.

Demonstrations were collected from either a human expert or an agent executing these tasks. No new benchmarks were introduced; rather, DRoP was compared against existing methods on these established environments.

Key Insights and Ideas

  • Online Confidence is Crucial: The paper highlights that dynamically updating confidence based on the agent's own experience is more effective than static or offline confidence measures. This allows the agent to learn when to trust and when to ignore demonstrations.
  • Surpassing the Demonstrator: By adaptively balancing prior knowledge with self-learning, DRoP enables agents to not only learn faster but also to eventually outperform the original demonstrations, a limitation of many imitation learning approaches.
  • Intelligent Demonstration Requests: The confidence measure can be used to identify states where the agent is uncertain. This allows DRoP to intelligently request additional demonstrations specifically in these critical areas, leading to more efficient learning than simply collecting more data indiscriminately.
  • Handling Suboptimal Data: DRoP demonstrates robustness to suboptimal or noisy demonstrations. The confidence mechanism allows the agent to effectively "learn to ignore" poor quality data, preventing it from hindering overall performance.

Key Results

  • Superior Performance: DRoP consistently outperformed baseline methods (Q-learning, HAT, Dagger, CHAT) in both Cartpole and Mario domains across metrics like jumpstart (initial performance), total reward, and final converged performance. For instance, in Cartpole, DRoP achieved a final reward of 2411 ± 56, significantly higher than Q-learning's 951 ± 36.
  • Effective Reuse of Low-Quality Data: When presented with demonstrations of varying quality (from random to high-performing), DRoP was able to correctly leverage the highest-quality data while effectively ignoring the lower-quality data, as evidenced by its reuse frequency.
  • Efficient Interactive Learning: Requesting demonstrations in low-confidence states significantly improved learning efficiency. The time cost for collecting these targeted demonstrations was a small fraction (2%) of the baseline training time, yet yielded substantial performance gains.

Limitations and Future Work

  • Domain Variance: The paper notes that the "neighborhood window" size used for identifying states to request demonstrations from might be domain-variant.
  • Model-Based Demonstrations: The authors suggest exploring model-based demonstrations, which might offer better confidence measurements than classification-based approaches.
  • Lifelong Learning: Implementing DRoP as a lifelong online learning system to continuously refine knowledge models is a future direction.
  • Bootstrap for Deep Learning: Investigating DRoP as a bootstrap mechanism for light-weight, pre-trained knowledge networks from demonstrations is also proposed, especially considering the training costs of deep learning.

Summary updated

Abstract

Reinforcement learning has enjoyed multiple impressive successes in recent years. However, these successes typically require very large amounts of data before an agent achieves acceptable performance. This paper focuses on a novel way of combating such requirements by leveraging existing (human or agent) knowledge. In particular, this paper leverages demonstrations, allowing an agent to quickly achieve high performance. This paper introduces the Dynamic Reuse of Prior (DRoP) algorithm, which combines the offline knowledge (demonstrations recorded before learning) with online confidence-based performance analysis. DRoP leverages the demonstrator's knowledge by automatically balancing between reusing the prior knowledge and the current learned policy, allowing the agent to outperform the original demonstrations. We compare with multiple state-of-the-art learning algorithms and empirically show that DRoP can achieve superior performance in two domains. Additionally, we show that this confidence measure can be used to selectively request additional demonstrations, significantly improving the learning performance of the agent.
BibTeX
@inproceedings{wang2019interactive,
  author = {Zhaodong Wang and Matthew E. Taylor},
  title = {{Interactive Reinforcement Learning with Dynamic Reuse of Prior Knowledge from Human and Agent Demonstrations}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  url = {https://openreview.net/forum?id=eOAH6y7yMu},
}
Figure