Introduction
Reinforcement Learning (RL) has traditionally relied on environmental rewards to guide agent behavior. However, integrating human expertise into this process often encounters a fundamental mismatch: humans do not naturally provide consistent numerical reward signals. This research, presented at IJCAI 2015, proposes Policy Shaping as a more effective alternative. Instead of treating human feedback as a scalar reward, Policy Shaping interprets it as direct advice on action selection, yielding significant improvements in learning speed and robustness.
The Shift from Reward to Policy Shaping
In Interactive Machine Learning (IML), non-expert users often struggle to act as reliable reward functions. They may provide sparse feedback, exhibit a strong positivity bias, or cease feedback once an agent achieves a baseline level of competence. Traditional Reward Shaping incorporates this feedback as an additional term in the reward function, which can lead to unintended behaviors if the human's "reward" is inconsistent with the environment's goals.
Policy Shaping addresses this by treating human critiques as evaluations of specific actions (e.g., "that move was correct"). The agent maintains a probability $P_{rc}$ that an action is correct based on human feedback history, assuming the teacher is correct with a fixed probability $C$ (set to 0.7 for these experiments). This human-derived probability is then multiplied by the agent's internal RL action probability $P_{rq}$ (derived from Q-learning with Boltzmann exploration) to determine the final policy.
Experimental Methodology
The researchers evaluated their approach using a custom Pac-Man environment. Twenty-six human participants were tasked with teaching an RL agent by critiquing its actions in pre-recorded gameplay videos. To ensure experimental control and test the impact of instructional bias, the study utilized several board configurations, including mirrored versions of the same states to prevent participants from recognizing repeated scenarios.
Participants were divided into groups receiving different instructions regarding how their silence would be interpreted:
- Open Ended: No specific instructions on the meaning of silence.
- Biased Good: Told that silence would be interpreted as "good."
- Biased Bad: Told that silence would be interpreted as "bad."
The Signal in Silence
A key finding of the study was the importance of interpreting teacher silence, which occurred approximately 30% of the time. The researchers tested an "Assumed Good" strategy, where the algorithm treats silence as a positive critique. In the Pac-Man domain—where most random actions lead to failure—any action that does not result in immediate death is relatively "good." Consequently, assuming silence is positive significantly boosted performance across all experimental groups, regardless of the instructions given to the teachers.
Results and the "Rigid Oracle"
The most striking result was that real human teachers outperformed a simulated "Oracle" teacher. While the Oracle followed a single optimal policy and penalized any deviation, human teachers recognized that Pac-Man allows for multiple winning strategies. By not discouraging alternative successful paths, humans provided a richer, more flexible training signal than the perfect but rigid simulated baseline.
As shown in the performance graph, the "Open Ended" group consistently reached higher average scores faster than groups given specific biases. This suggests that human intuition is most effective when not constrained by rigid mathematical interpretations of their feedback or silence. Trying to "bias" humans by telling them how their silence would be interpreted actually decreased the quality of the data provided.
Conclusion
Policy Shaping offers a robust framework for incorporating human feedback into RL by treating humans as action critics rather than reward sources. By carefully interpreting the meaning of silence and allowing for multiple valid policies, agents can learn more efficiently than from perfect but rigid oracles. Future work will focus on developing agents that can autonomously determine the meaning of a specific teacher's silence or tailor their interpretation to individual teaching styles.
Summary updated