Reinforcement Learning from Demonstration through Shaping

Tim Brys, Anna Harutyunyan, Halit Bener Suay, Sonia Chernova, Matthew E. Taylor, Ann NowéIJCAI 2015 · OpenReview

Abstract

Reinforcement learning describes how a learning agent can achieve optimal behaviour based on interactions with its environment and reward feedback. A limiting factor in reinforcement learning as employed in artificial intelligence is the need for an often prohibitively large number of environment samples before the agent reaches a desirable level of performance. Learning from demonstration is an approach that provides the agent with demonstrations by a supposed expert, from which it should derive suitable behaviour. Yet, one of the challenges of learning from demonstration is that no guarantees can be provided for the quality of the demonstrations, and thus the learned behavior. In this paper, we investigate the intersection of these two approaches, leveraging the theoretical guarantees provided by reinforcement learning, and using expert demonstrations to speed up this learning by biasing exploration through a process called reward shaping. This approach allows us to leverage human input without making an erroneous assumption regarding demonstration optimality. We show experimentally that this approach requires significantly fewer demonstrations, is more robust against suboptimality of demonstrations, and achieves much faster learning than the recently developed HAT algorithm.

Introduction

Reinforcement Learning (RL) has demonstrated remarkable success in solving complex control tasks through autonomous exploration. However, its practical application is often hindered by extreme sample inefficiency, particularly in environments with sparse rewards. Conversely, Learning from Demonstration (LfD) offers a shortcut by allowing agents to mimic expert behavior. The primary drawback of LfD is its sensitivity to demonstration quality; if a teacher provides suboptimal or inconsistent data, the agent's performance is typically capped by these flaws.

This research proposes a hybrid framework, RLfD through Shaping, which utilizes demonstrations as a heuristic bias to guide RL exploration rather than a rigid template for mimicry. By integrating demonstrations into a potential-based reward shaping function, the agent can benefit from expert hints while maintaining the ability to converge to an optimal policy, even if the demonstrations are flawed.

Methodology: Reward Shaping from Demonstrations

The core of the proposed method is the conversion of raw demonstration data into a continuous reward signal that biases the agent toward "expert-like" states.

1. Similarity Metric and Potential Function

The authors define a similarity function based on a multi-variate Gaussian. For any state-action pair $(s, a)$ encountered by the agent, its similarity to the demonstration set $D$ is calculated. The potential function $\Phi^D(s, a)$ is defined as the maximum similarity between the current pair and any pair $(s^d, a)$ in the demonstration set. This creates a "potential landscape" where regions near demonstrated trajectories have high values, effectively guiding the agent's exploration.

2. Potential-Based Reward Shaping (PBRS)

To ensure that this bias does not distort the final learned behavior, the authors employ Potential-Based Reward Shaping. The additional reward $F$ is defined as $F = \gamma\Phi(s', a') - \Phi(s, a)$. Theoretically, PBRS is proven to preserve the optimal policy of the original environment. This means that while the shaping signal accelerates learning, the agent will eventually prioritize the environment's true rewards, allowing it to "outperform" a suboptimal teacher.

3. Q-Value Initialization

In addition to the dynamic reward signal, the potential function is used to initialize the agent's Q-values. This ensures that the heuristic bias influences the agent's decisions from the very first episode of training.

Experimental Results

The framework was evaluated on the Cart Pole balancing task and the Mario AI Benchmark.

Learning Acceleration in Cart Pole

In the Cart Pole environment, RLfD through shaping demonstrated a dramatic increase in learning speed. As shown in Figure 1, the proposed method reached near-optimal performance (balancing for 1000 steps) in approximately 200 episodes. In contrast, standard RL required nearly 800 episodes to reach a similar level, and the HAT (Heuristic Acceleration Transfer) algorithm showed significantly slower progress.

Learning curves in the Cart Pole environment with 128 demonstration samples.

Robustness to Suboptimal Demonstrators

A key strength of the shaping approach is its robustness to the quality of the teacher. In the Mario domain, the authors tested the agent using demonstrations from four sources: suboptimal RL agents, hand-coded agents, human players, and optimal RL agents.

Performance in the Mario environment across different demonstrator types.

As illustrated in Figure 2, RLfD through shaping consistently outperformed standard RL and HAT across all demonstrator types. Notably, even when provided with suboptimal demonstrations (e.g., scoring only ~960 points), the RLfD agent eventually reached the same high final performance (~2400 points) as an agent trained with optimal demonstrations. This is because the Gaussian-based shaping provides a "local" bias that rewards being near specific successful points, whereas classifier-based methods like HAT often struggle with the inconsistencies inherent in human data.

Conclusion

The results indicate that RLfD through shaping provides a powerful middle ground between pure RL and pure LfD. By treating demonstrations as a non-binding heuristic, the method achieves extreme sample efficiency—showing significant gains with as few as two demonstration samples in some cases—without sacrificing the theoretical guarantees of RL. Future work may address the computational overhead of calculating potentials in large demonstration sets and explore ensemble methods to further enhance learning stability.

Summary updated

Abstract

Reinforcement learning describes how a learning agent can achieve optimal behaviour based on interactions with its environment and reward feedback. A limiting factor in reinforcement learning as employed in artificial intelligence is the need for an often prohibitively large number of environment samples before the agent reaches a desirable level of performance. Learning from demonstration is an approach that provides the agent with demonstrations by a supposed expert, from which it should derive suitable behaviour. Yet, one of the challenges of learning from demonstration is that no guarantees can be provided for the quality of the demonstrations, and thus the learned behavior. In this paper, we investigate the intersection of these two approaches, leveraging the theoretical guarantees provided by reinforcement learning, and using expert demonstrations to speed up this learning by biasing exploration through a process called reward shaping. This approach allows us to leverage human input without making an erroneous assumption regarding demonstration optimality. We show experimentally that this approach requires significantly fewer demonstrations, is more robust against suboptimality of demonstrations, and achieves much faster learning than the recently developed HAT algorithm.
BibTeX
@inproceedings{brys2015reinforcement,
  author = {Tim Brys and Anna Harutyunyan and Halit Bener Suay and Sonia Chernova and Matthew E. Taylor and Ann Nowé},
  title = {{Reinforcement Learning from Demonstration through Shaping}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2015},
  url = {https://openreview.net/forum?id=S145CVGuWB},
}
Figure