Reinforcement Learning with Human Teachers: Evidence of Feedback and Guidance with Implications for Learning Performance

Andrea Lockerd Thomaz, Cynthia BreazealAAAI 2006

Abstract

Traditional Reinforcement Learning (RL) frameworks typically treat human input as a simple reward signal for past actions. This paper identifies a fundamental mismatch: human teachers naturally attempt to provide "anticipatory rewards" to guide future behavior. By introducing a dedicated guidance channel into the RL algorithm, the authors demonstrate that agents can learn significantly faster and more efficiently. Experimental results in a simulated kitchen environment show a ~50% reduction in the number of trials and states visited when human guidance is leveraged.

Introduction: The Feedback-Guidance Mismatch

Reinforcement Learning often suffers from the "curse of dimensionality," requiring extensive exploration to find optimal policies. While "human-in-the-loop" systems aim to accelerate this process, they often fail to account for the social nature of human teaching. Most RL algorithms expect feedback—a critique of a previous action. However, human teachers frequently intend to provide guidance—advice on what the agent should do next. When an algorithm interprets guidance as feedback, it creates a "socially awkward" interaction that slows learning and frustrates the teacher.

Methodology

The study utilized Sophie’s Kitchen, a Java-based simulation where a virtual agent (Sophie) learns to bake a cake. The environment consists of approximately 10,000 states involving objects like flour, eggs, and an oven.

The researchers conducted an initial observational study with 18 non-experts to identify natural teaching patterns. Based on these findings, they developed Interactive Q-Learning with Guidance. In this modified algorithm, the teacher can provide standard scalar rewards (feedback) or use a "pre-action" phase to highlight specific objects (guidance). If a teacher highlights an object, the agent prunes its exploration tree, considering only actions relevant to that object.

Key Insights: Anticipatory Rewards

A critical finding from the observational study was the prevalence of "anticipatory rewards." Participants frequently rewarded the agent for interacting with objects it hadn't used yet—for example, rewarding an empty bowl to suggest the agent should pick it up.

Human teachers naturally provide anticipatory rewards to empty objects to guide future actions.

In standard RL, these signals are treated as noise or incorrect feedback. However, in a human-centric framework, they represent clear instructions. By allowing the human to "point" at objects, the agent uses the human as a high-level heuristic, avoiding irrelevant actions like trying to use the oven before gathering ingredients.

Results and Performance Gains

The inclusion of a guidance channel led to dramatic improvements in learning efficiency for both expert and non-expert teachers. The agent was able to reach the goal state much faster while avoiding "disaster" states that typically lead to failed trials.

Comparative performance metrics between non-guided and guided learning sessions.

Key quantitative improvements included:

  • Learning Speed: A 49% reduction in the number of trials and a 55% reduction in total actions required to learn the task.
  • Exploration Efficiency: Guided agents visited 50% fewer unique states, showing that human attention effectively prunes the search space.
  • Reliability: The number of failed trials before the first success dropped by 41%.
  • State Quality: Guided agents spent 72.4% of their time in "good" states, compared to 60.3% for those receiving only feedback.

Conclusion and Future Work

This research demonstrates that RL algorithms are more effective when they align with natural human teaching behaviors. By treating the human as a guide rather than just a reward function, agents can overcome the curse of dimensionality more effectively. Future work will focus on scaling these methods to continuous, real-world robotic environments and exploring more natural communication modalities, such as speech and gesture, to further tighten the human-agent loop.

Summary updated

BibTeX
@inproceedings{thomaz2006reinforcement,
  author = {Andrea Lockerd Thomaz and Cynthia Breazeal},
  title = {{Reinforcement Learning with Human Teachers: Evidence of Feedback and Guidance with Implications for Learning Performance}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2006},
}
Figure