Policy Shaping with Human Teachers

Thomas Cederborg, Ishaan Grover, Charles L. Isbell Jr., Andrea Lockerd ThomazIJCAI 2015

Introduction

Reinforcement Learning (RL) has traditionally relied on environmental rewards to guide agent behavior. However, integrating human expertise into this process often encounters a fundamental mismatch: humans do not naturally provide consistent numerical reward signals. This research, presented at IJCAI 2015, proposes Policy Shaping as a more effective alternative. Instead of treating human feedback as a scalar reward, Policy Shaping interprets it as direct advice on action selection, yielding significant improvements in learning speed and robustness.

The Shift from Reward to Policy Shaping

In Interactive Machine Learning (IML), non-expert users often struggle to act as reliable reward functions. They may provide sparse feedback, exhibit a strong positivity bias, or cease feedback once an agent achieves a baseline level of competence. Traditional Reward Shaping incorporates this feedback as an additional term in the reward function, which can lead to unintended behaviors if the human's "reward" is inconsistent with the environment's goals.

Policy Shaping addresses this by treating human critiques as evaluations of specific actions (e.g., "that move was correct"). The agent maintains a probability $P_{rc}$ that an action is correct based on human feedback history, assuming the teacher is correct with a fixed probability $C$ (set to 0.7 for these experiments). This human-derived probability is then multiplied by the agent's internal RL action probability $P_{rq}$ (derived from Q-learning with Boltzmann exploration) to determine the final policy.

Experimental Methodology

The researchers evaluated their approach using a custom Pac-Man environment. Twenty-six human participants were tasked with teaching an RL agent by critiquing its actions in pre-recorded gameplay videos. To ensure experimental control and test the impact of instructional bias, the study utilized several board configurations, including mirrored versions of the same states to prevent participants from recognizing repeated scenarios.

The experimental setup involved a primary training board and several evaluation boards (v1-v3) including mirrored versions (v2m, v3m) to test instructional consistency across identical state-action pairs.

Participants were divided into groups receiving different instructions regarding how their silence would be interpreted:

  1. Open Ended: No specific instructions on the meaning of silence.
  2. Biased Good: Told that silence would be interpreted as "good."
  3. Biased Bad: Told that silence would be interpreted as "bad."

The Signal in Silence

A key finding of the study was the importance of interpreting teacher silence, which occurred approximately 30% of the time. The researchers tested an "Assumed Good" strategy, where the algorithm treats silence as a positive critique. In the Pac-Man domain—where most random actions lead to failure—any action that does not result in immediate death is relatively "good." Consequently, assuming silence is positive significantly boosted performance across all experimental groups, regardless of the instructions given to the teachers.

Results and the "Rigid Oracle"

The most striking result was that real human teachers outperformed a simulated "Oracle" teacher. While the Oracle followed a single optimal policy and penalized any deviation, human teachers recognized that Pac-Man allows for multiple winning strategies. By not discouraging alternative successful paths, humans provided a richer, more flexible training signal than the perfect but rigid simulated baseline.

Learning curves comparing different instructional biases. The 'Open Ended' group achieved the highest performance, suggesting that forcing humans into specific feedback models can be counterproductive.

As shown in the performance graph, the "Open Ended" group consistently reached higher average scores faster than groups given specific biases. This suggests that human intuition is most effective when not constrained by rigid mathematical interpretations of their feedback or silence. Trying to "bias" humans by telling them how their silence would be interpreted actually decreased the quality of the data provided.

Conclusion

Policy Shaping offers a robust framework for incorporating human feedback into RL by treating humans as action critics rather than reward sources. By carefully interpreting the meaning of silence and allowing for multiple valid policies, agents can learn more efficiently than from perfect but rigid oracles. Future work will focus on developing agents that can autonomously determine the meaning of a specific teacher's silence or tailor their interpretation to individual teaching styles.

Summary updated

BibTeX
@inproceedings{cederborg2015policy,
  author = {Thomas Cederborg and Ishaan Grover and Charles L. Isbell Jr. and Andrea Lockerd Thomaz},
  title = {{Policy Shaping with Human Teachers}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2015},
}
Figure