Improving Reinforcement Learning with Human Input

Matthew E. TaylorIJCAI 2018 · DOI

TL;DR

This paper explores how to make Reinforcement Learning (RL) agents learn more effectively by leveraging input from non-technical humans. It presents three key ways humans can help: providing demonstrations, offering evaluative feedback, and designing learning curricula, demonstrating that human guidance can significantly speed up learning and improve performance.

What Problem Does This Paper Address?

Reinforcement learning (RL) agents can achieve impressive results, but often require vast amounts of data and computation to reach a desired performance level. This is particularly problematic in real-world applications where poor performance can have real costs, or when defining an accurate reward function is difficult for a non-expert user. This paper tackles the challenge of how to effectively utilize the knowledge of a non-technical human, who might not be an expert but can still provide valuable guidance, to accelerate and improve the learning process of an RL agent.

Methods

The paper outlines three primary methods for incorporating human input into RL:

  • Leveraging Human Demonstrations: This approach reframes the learning problem as mimicking a demonstrator.
  • Human-Agent Transfer (HAT): This method uses demonstrations to create an initial bias for the RL agent. It learns a policy from human demonstrations and uses it to bootstrap RL, gradually reducing reliance on the demonstration as the agent learns its own value function.
  • Confidence-Based HAT (CHAT): An extension of HAT, CHAT also learns a confidence measure for the demonstrated actions. The agent only uses the demonstrated action if its confidence in the predicted action exceeds a certain threshold, leading to more robust learning.
  • Shaping with Inverse Reinforcement Learning (IRL): Instead of directly learning a policy, this method uses human demonstrations to learn a reward function. This learned reward function is then used to shape the agent's existing reward signal, guiding it towards better behavior without altering the optimal policy.
  • Leveraging Human Feedback (SABL): This method assumes no environmental reward is available. Instead, a human trainer provides explicit positive or negative feedback on the agent's actions. The Strategy-Aware Bayesian Learning (SABL) algorithm models the trainer's feedback strategy, including the probability of mistakes and the likelihood of providing feedback, allowing the agent to learn even when feedback is sparse or implicit.
  • Leveraging Human Curricula: Humans naturally learn by starting with simpler concepts and progressing to more complex ones. This approach applies curriculum learning to RL, where humans design a sequence of tasks for the agent to learn. The paper explores how non-expert humans can create these curricula, and how to bias the learning process based on the complexity and relevance of tasks in the human-designed curriculum.

Datasets

The paper discusses experiments conducted in several domains:

  • Keepaway Soccer: A multi-agent RL domain used to evaluate HAT methods.
  • Mario: A game used to evaluate CHAT.
  • Contextual Bandit Setting: Used for studies involving SABL and human-subject experiments.
  • Maze Domain: Used to evaluate shaping with IRL.
  • Sokoban-like Environment: Used for implicit communication experiments.
  • A Library of 16 Environments: Used for human-generated curricula experiments.

The paper does not introduce a new benchmark dataset but rather uses existing or simulated environments to test the proposed methods.

Key Insights and Ideas

  • Human intuition is valuable, even from non-experts: The paper strongly suggests that even sub-optimal human guidance can significantly improve RL agent performance and learning speed.
  • Implicit feedback carries meaning: The SABL algorithm highlights that the absence of feedback can be as informative as explicit feedback, depending on the trainer's strategy.
  • Curriculum design matters: While human intuition for curriculum design is beneficial, simply providing tasks in increasing order of difficulty isn't always optimal. Biasing the curriculum based on task relevance to the final goal can be more effective.
  • Confidence is key for demonstrations: CHAT demonstrates that knowing when to trust a demonstrated action (based on confidence) leads to better performance than blindly following demonstrations.

Key Results

  • HAT methods consistently improved performance and reduced training time compared to learning from scratch in domains like Keepaway Soccer. For instance, one HAT learner saved approximately 7 hours of training time using only 3 minutes of human demonstration.
  • CHAT, by incorporating confidence thresholds, significantly improved performance over standard HAT, especially in scenarios where the demonstrator's policy might not always be optimal.
  • Shaping with IRL demonstrated improvements in learning speed in maze domains and Mario.
  • SABL showed that treating human feedback as categorical rather than numeric led to better performance in contextual bandit tasks.
  • Human-designed curricula, when biased towards concepts relevant to the target task, outperformed randomly generated curricula and improved learning speed.

Limitations and Future Work

  • When and why methods differ: The paper acknowledges that it's not always clear which demonstration-based learning method is superior for a given domain. Future work could investigate the factors that make one method better than another.
  • Multi-modal advice: The paper raises the question of what possibilities exist for combining different types of human advice (e.g., demonstrations, feedback, natural language) for richer guidance.
  • On-demand demonstrations: The authors suggest exploring scenarios where a human can provide demonstrations on demand, either proactively or when the agent requests help.
  • Human understanding of agents: The paper questions how to better help non-technical participants understand the internal workings of agents, and whether this improved understanding leads to better advice and more effective learning.
  • Domain-specific expertise: The paper notes that while human intuition is valuable, it's unclear how domain-specific knowledge from humans could be further leveraged.

Summary updated

BibTeX
@inproceedings{taylor2018improving,
  author = {Matthew E. Taylor},
  title = {{Improving Reinforcement Learning with Human Input}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.24963/ijcai.2018/817},
}
Figure