Active Advice Seeking for Inverse Reinforcement Learning

Phillip Odom, Sriraam NatarajanAAAI 2015 · DOI

Abstract

Inverse Reinforcement Learning (IRL) traditionally relies on extensive expert demonstrations or manually selected advice to recover an agent's reward function. This paper proposes an Active Advice Seeking framework where the agent proactively identifies states of high uncertainty to query an expert for targeted guidance. By combining demonstration and policy uncertainty, the approach significantly reduces human workload while improving learning efficiency in complex environments.

Introduction

Standard IRL attempts to infer an underlying reward function by observing an expert's behavior. However, collecting a sufficient volume of near-perfect demonstrations is often prohibitively expensive. While "advice"—general rules or constraints on behavior—can alleviate this, previous methods required the human expert to identify where advice was most needed. This paper introduces an active learning loop that empowers the agent to pinpoint its own confusion, asking for advice only where it will be most impactful.

Methodology: Active Advice Seeking

The core of the proposed framework is an iterative algorithm that calculates a state-specific uncertainty measure $u(x)$ to guide expert queries. Uncertainty is derived from two primary sources:

  1. Demonstration Uncertainty ($f(x)$): The entropy of expert actions in a given state. High entropy suggests the agent lacks clear evidence of the "correct" behavior from the provided demonstrations.
  2. Policy Uncertainty ($g(x)$): The entropy of the learned policy (e.g., via a Boltzmann distribution over Q-values). This captures instances where the current reward function makes multiple actions appear equally viable.

The total uncertainty is a weighted sum: $u(x) = w_p f(x) + (1 - w_p) g(x)$. The agent iteratively selects the $N$ states with the highest $u(x)$, receives expert advice for those states, and updates its reward function.

Experimental Setup

The authors evaluated the algorithm in the WumpusWorld domain, a discrete grid-based environment where an agent must navigate from a start (S) to a goal (G) while avoiding pits and monsters.

The WumpusWorld environment showing start (S), goal (G), and regions where advice (A1, A2) might be provided.

The agent begins with a set of potentially sub-optimal demonstrations. "Advice" in this context is more expressive than simple labels; it can define sets of preferred actions or general constraints that generalize across the state space.

Results and Discussion

The active uncertainty-based strategy was compared against random advice sampling and a baseline using only initial trajectories.

Win rate percentages for different advice-seeking strategies across varying amounts of advice.

As shown in the results, the Uncertainty-based method scales significantly better than random sampling. With 10 pieces of advice, the active seeker achieved a 33.3% win rate, a substantial improvement over the 9.9% achieved by random advice and the 0.8% baseline from demonstrations alone. While random sampling showed a slight advantage at very low advice counts (1-3), the active strategy quickly overtook it as the agent gathered enough information to accurately identify its own knowledge gaps.

Conclusion and Future Work

Active Advice Seeking effectively addresses the "Many Rewards, One Policy" problem in IRL by distinguishing between reward and action uncertainty. By focusing human effort on the most ambiguous states, the framework reduces cognitive load and improves performance. Future research will explore scaling this approach to continuous and factored state spaces, as well as developing methods to automatically determine the optimal volume of advice required for a specific task.

Summary updated

BibTeX
@inproceedings{odom2015active,
  author = {Phillip Odom and Sriraam Natarajan},
  title = {{Active Advice Seeking for Inverse Reinforcement Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2015},
  doi = {10.1609/aaai.v29i1.9722},
}
Figure