Theoretically-Grounded Policy Advice from Multiple Teachers in Reinforcement Learning Settings with Applications to Negative Transfer

Yusen Zhan, Haitham Bou-Ammar, Matthew E. Taylor

Executive Summary

This research establishes the first formal theoretical framework for action advice in Reinforcement Learning (RL) involving multiple, potentially imperfect teachers. By introducing a "Grand-Teacher" meta-policy and a novel mixing algorithm, the authors provide mathematical regret bounds that quantify how teacher quality impacts student learning. Crucially, the framework identifies the conditions for "negative transfer"—where bad advice hinders progress—and demonstrates how a student can eventually surpass sub-optimal teachers to reach an optimal policy.

The Challenge: Imperfect Advice and Negative Transfer

Reinforcement Learning is often limited by sample inefficiency, requiring extensive trial-and-error. While "policy advice" (where a teacher suggests actions) has shown empirical success in speeding up learning, it has historically lacked a rigorous theoretical foundation. Most existing methods assume teachers are optimal, but real-world teachers are often flawed. If a student blindly follows a sub-optimal teacher, it may learn slower than if it had no teacher at all, a phenomenon known as negative transfer. This paper moves policy advice from an empirical heuristic to a theoretically-grounded technique capable of handling multiple, imperfect sources of information.

Methodology: The Grand-Teacher Framework

The authors propose a multi-teacher framework centered on two key components:

  • The Grand-Teacher: A meta-policy that aggregates advice from multiple teachers. This can be constructed online (querying teachers as new states are encountered) or offline (pre-calculating a majority-vote policy).
  • Algorithm 3: This core algorithm blends the Grand-Teacher’s advice with the student’s own exploration. It utilizes a mixing parameter ($\beta$) that dictates the level of trust in the teacher. This parameter decays over time, allowing the student to rely on advice early in the learning process but eventually transition to its own learned policy, optimized via the REGAL.C algorithm.

Through formal regret analysis, the authors derive bounds that measure the cost of learning, allowing for a direct comparison between the student's performance and an optimal strategy.

Experimental Benchmarks

The approach was evaluated on three classic RL environments designed to test exploration and sequence-dependent decision-making:

  • Combination Lock: A sensitive environment where a single error resets progress, making exploration nearly impossible without guidance.
  • Grid World: A four-room navigation task with stochastic movement.
  • Block Dude: A complex puzzle requiring non-obvious sequences of actions to reach a goal.
State-transition diagram for the Combination Lock environment, illustrating the difficulty of reaching the goal state 'n' without specific action sequences.

Key Results and Performance

Empirical results across all benchmarks confirmed the theoretical predictions. The proposed method demonstrated several key advantages:

  • Surpassing Sub-optimal Teachers: Unlike previous methods (such as Azar’s) that often converge to the teacher's performance level, this algorithm allows the student to continue refining its policy until it reaches the true optimum, even when starting with "worst-case" or random advice.
  • Mitigating Negative Transfer: The framework provides a mathematical condition to identify when source knowledge will hinder the student, enabling the development of safer transfer learning systems.
  • Superiority over Baselines: The method consistently outperformed Azar’s method, which failed to improve in these sparse-reward settings, remaining at the baseline reward level.
Average reward across three benchmarks. The results show that while good advice accelerates learning, bad advice (Worst/Rand Teacher) leads to negative transfer, though the student eventually recovers to the optimal policy.

Theoretical Insights

The paper proves that a teacher's "regret ratio" determines their utility: if a teacher is better than a random baseline, learning speed improves; if worse, regret increases. This formalization allows researchers to quantify the "better than constant" improvement that advice provides over standard learners.

Limitations and Future Work

While providing a strong theoretical foundation, the work has limitations. The "Offline Grand-Teacher" can be computationally expensive in large state spaces due to exploration requirements. Additionally, the current theory is specifically tied to the REGAL.C algorithm. Future research aims to adapt these proofs for other popular RL algorithms like R-max or Deep RL settings involving continuous, high-dimensional state spaces.

Summary updated

Abstract

Policy advice is a transfer learning method where a student agent is able to learn faster via advice from a teacher. However, both this and other reinforcement learning transfer methods have little theoretical analysis. This paper formally defines a setting where multiple teacher agents can provide advice to a student and introduces an algorithm to leverage both autonomous exploration and teacher's advice. Our regret bounds justify the intuition that good teachers help while bad teachers hurt. Using our formalization, we are also able to quantify, for the first time, when negative transfer can occur within such a reinforcement learning setting.
BibTeX
@inproceedings{zhan2016theoretically,
  author = {Yusen Zhan and Haitham Bou-Ammar and Matthew E. Taylor},
  title = {{Theoretically-Grounded Policy Advice from Multiple Teachers in Reinforcement Learning Settings with Applications to Negative Transfer}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2016},
}
Figure