TL;DR
In interactive reinforcement learning, we often assume that the best "teacher" is the agent with the highest score. This paper reveals the "Expert Paradox": a high-performing specialist often makes a poor teacher because they lack knowledge of non-optimal states. Instead, "polymath" agents—those with broad, evenly distributed experience across the entire task space—provide superior guidance, leading to faster convergence and more stable learning.
What Problem Does This Paper Address?
Reinforcement Learning (RL) is powerful but notoriously slow, as agents must explore vast state spaces through trial and error. Interactive Reinforcement Learning (IRL) speeds this up by introducing a "teacher" (human or artificial) to provide advice.
However, a critical question remains: What actually makes a good teacher? Most researchers simply pick the agent with the highest accumulated reward. The authors argue this is flawed. A specialist who only knows the "perfect path" to a goal cannot help a student who wanders off-track. The paper investigates how the internal knowledge representation and experience distribution of a teacher-agent affect the success of a learner-agent.
Methods
The researchers used a simulated Domestic Robot Scenario where a robot must clean a table by moving a cup and using a sponge. They employed the SARSA algorithm (an on-policy RL method) and compared three types of artificial teachers:
- Specialist-A: Optimized for the shortest path; high reward but narrow experience.
- Specialist-B: Follows a longer, alternative path.
- Polymath: Explores all possible states and paths more evenly, characterized by a low standard deviation in state-visit frequency.
The study also evaluated three interaction parameters:
- Probability of Feedback ($L$): How often the teacher offers advice.
- Consistency of Feedback ($C$): The likelihood that the advice is actually the optimal action.
- Learner Obedience ($O$): How often the learner actually follows the advice versus doing its own thing.
Datasets
The study does not use a standard external dataset. Instead, it utilizes a custom-designed Domestic Robot Scenario benchmark:
- State Space: 53 discrete states representing the robot's hand position, the object held, the cup's location, and the table's cleanliness.
- Action Space: 7 domain-specific actions (GET, DROP, CLEAN, etc.).
- Evaluation: Results were averaged over 100 independent agents trained over 3,000 episodes each.
Key Insights and Ideas
- The Expert Paradox: An agent can be an "expert" at performing a task while being a "novice" at teaching it. Specialist-A had the highest rewards but left "knowledge gaps" (represented as cold spots in Q-value heatmaps) in states outside its preferred path.
- The Polymath Advantage: The polymath agent, despite having a lower individual reward during its own training, was a significantly better teacher. Because it had visited almost every state, it could provide helpful advice no matter where the learner-agent wandered.
- Consistency Trumps Frequency: Increasing the amount of advice ($L$) doesn't help much if the advice is inconsistent. However, even small improvements in the quality (consistency $C$) of advice lead to massive jumps in learner performance.
- Stable Behavior: Learners guided by polymaths showed much lower variance in their behavior. They didn't just learn faster; they learned more reliably.
Key Results
- Convergence Speed: Learners with polymath teachers reached peak reward significantly faster than those with specialist teachers or those learning autonomously.
- Reward Magnitude: Using a polymath advisor led to higher overall accumulated rewards for the learner compared to using a specialist-A advisor.
- Stability: The standard deviation of state visits for IRL agents was much lower when using a polymath teacher, indicating a more "stable" and predictable learning trajectory.
- Sensitivity to "Bad" Advice: The researchers found that even a 5-10% drop in advice consistency (from 0.95 to 0.85) could make the difference between a learner that outperforms autonomous RL and one that performs worse than if it had no teacher at all.
Limitations and Future Work
- Discrete vs. Continuous: The current study is limited to a small, discrete state space. The authors acknowledge that scaling this to continuous environments (where most real-world robotics tasks live) is a major challenge, as discretizing continuous space can lead to mapping errors between teacher and student.
- Static Obedience: In this paper, "learner obedience" was a fixed parameter. The authors suggest that future learners should be adaptive—trusting the teacher implicitly at the start of training but gradually relying more on their own experience as they become experts themselves.
- Human-in-the-loop: While this study used artificial agents as teachers to maintain control over variables, the ultimate goal is to apply these insights to how humans (who are often inconsistent teachers) interact with robots.
Summary updated