Improving interactive reinforcement learning: What makes a good teacher?

Francisco Cruz, Sven Magg, Yukie Nagai, Stefan Wermter

Abstract

Interactive reinforcement learning has become an important apprenticeship approach to speed up convergence in classic reinforcement learning problems. In this regard, a variant of interactive reinforcement learning is policy shaping which uses a parent-like trainer to propose the next action to be performed and by doing so reduces the search space by advice. On some occasions, the trainer may be another artificial agent which in turn was trained using reinforcement learning methods to afterward becoming an advisor for other learner-agents. In this work, we analyze internal representations and characteristics of artificial agents to determine which agent may outperform others to become a better trainer-agent. Using a polymath agent, as compared to a specialist agent, an advisor leads to a larger reward and faster convergence of the reward signal and also to a more stable behavior in terms of the state visit frequency of the learner-agents. Moreover, we analyze system interaction parameters in order to determine how influential they are in the apprenticeship process, where the consistency of feedback is much more relevant when dealing with different learner obedience parameters.

Introduction

Reinforcement Learning (RL) is a powerful paradigm for training autonomous agents, but it is often hampered by slow convergence as agents explore vast state spaces through trial and error. Interactive Reinforcement Learning (IRL) addresses this by introducing a "teacher"—either human or artificial—to provide advice and narrow the search space.

However, a fundamental question arises: What makes a good teacher? In many IRL studies, the "best" teacher is assumed to be the agent with the highest performance or reward. This paper challenges that assumption, identifying an "Expert Paradox": a high-performing specialist often makes a poor teacher because they lack knowledge of non-optimal states. This work demonstrates that "polymath" agents—those with broad, evenly distributed experience—provide superior guidance, leading to faster and more stable learning.

Experimental Setup: The Domestic Robot Scenario

The researchers evaluated their hypothesis using a simulated Domestic Robot Scenario. In this environment, a robot must clean a table by manipulating a cup and a sponge.

The Interactive Reinforcement Learning framework involving a trainer, an agent, and the environment.

The task is defined by a state space of 53 discrete states representing variables such as hand position, the object being held, cup location, and table cleanliness, and an action space of 7 domain-specific actions (e.g., GET, DROP, CLEAN). The SARSA algorithm was used for both the teachers and the learners.

Three types of artificial teachers were compared:

  1. Specialist-A: Optimized for the shortest path to the goal.
  2. Specialist-B: Optimized for a longer, alternative path.
  3. Polymath: An agent that explores all possible states and paths more evenly during its training phase.

The Expert Paradox and the Polymath Advantage

The study reveals that an agent can be an "expert" at performing a task while being a "novice" at teaching it. Specialist agents, while achieving high rewards, leave significant "knowledge gaps" in states outside their preferred optimal path. If a learner-agent wanders into these "cold spots," the specialist teacher cannot provide helpful advice.

Comparison of state-visit frequencies between Specialist and Polymath agents, highlighting the broader coverage of the Polymath.

In contrast, the Polymath agent visited almost every state during its training. While it might have a lower individual reward than a specialist, its broad internal representation allows it to guide a learner from any position in the state space.

Key results include:

  • Faster Convergence: Learners guided by polymaths reached peak rewards significantly faster than those guided by specialists or those learning autonomously.
  • Higher Rewards: Polymath-guided learners achieved higher overall accumulated rewards.
  • Behavioral Stability: The variance in state visits for learners was much lower with a polymath teacher, indicating a more predictable and reliable learning trajectory.

Interaction Dynamics

Beyond the teacher's knowledge, the study examined three critical interaction parameters: Probability of Feedback ($L$), Consistency of Feedback ($C$), and Learner Obedience ($O$). The findings suggest that consistency trumps frequency. Increasing the amount of advice ($L$) provides diminishing returns if the advice is inconsistent. However, even marginal improvements in the quality (consistency $C$) of advice lead to substantial performance gains for the learner.

Conclusion

This research highlights that the effectiveness of a teacher in IRL is determined more by the breadth of their experience than by their peak performance. Polymath agents, by virtue of their comprehensive state-space coverage, are more resilient and effective advisors than specialists. Future work will explore scaling these concepts to continuous state spaces and developing adaptive learners that can dynamically adjust their reliance on teacher advice as they gain expertise.

Summary updated

Abstract

Interactive reinforcement learning has become an important apprenticeship approach to speed up convergence in classic reinforcement learning problems. In this regard, a variant of interactive reinforcement learning is policy shaping which uses a parent-like trainer to propose the next action to be performed and by doing so reduces the search space by advice. On some occasions, the trainer may be another artificial agent which in turn was trained using reinforcement learning methods to afterward becoming an advisor for other learner-agents. In this work, we analyze internal representations and characteristics of artificial agents to determine which agent may outperform others to become a better trainer-agent. Using a polymath agent, as compared to a specialist agent, an advisor leads to a larger reward and faster convergence of the reward signal and also to a more stable behavior in terms of the state visit frequency of the learner-agents. Moreover, we analyze system interaction parameters in order to determine how influential they are in the apprenticeship process, where the consistency of feedback is much more relevant when dealing with different learner obedience parameters.
BibTeX
@inproceedings{cruz2018improving,
  author = {Francisco Cruz and Sven Magg and Yukie Nagai and Stefan Wermter},
  title = {{Improving interactive reinforcement learning: What makes a good teacher?}},
  booktitle = {Connect. Sci},
  year = {2018},
  url = {https://openreview.net/forum?id=iSSywTfa4T},
}
Figure