Abstract
Inverse Reinforcement Learning (IRL) is the problem of learning the reward function underlying a Markov Decision Process given the dynamics of the system and the behaviour of an expert. IRL is motivated by situations where knowledge of the rewards is a goal by itself (as in preference elicitation) and by the task of apprenticeship learning (learning policies from an expert). In this paper we show how to combine prior knowledge and evidence from the expert's actions to derive a probability distribution over the space of reward functions. We present efficient algorithms that find solutions for the reward learning and apprenticeship learning tasks that generalize well over these distributions. Experimental results show strong improvement for our methods over previous heuristic-based approaches.
Introduction
In the standard Reinforcement Learning (RL) paradigm, an agent seeks to maximize a predefined reward function. Inverse Reinforcement Learning (IRL) reverses this objective: given an expert's behavior, the goal is to recover the underlying reward function. This problem is inherently ill-posed, as multiple reward functions can often explain the same optimal behavior. Traditional approaches have typically relied on heuristics to select a single point estimate for the reward.
Ramachandran and Amir (2007) address these challenges by introducing Bayesian Inverse Reinforcement Learning (BIRL). This framework treats the expert's actions as evidence to update a probability distribution over the space of potential reward functions, allowing for a principled treatment of ambiguity, the integration of prior domain knowledge, and robustness to expert fallibility.
The Bayesian IRL Framework
The BIRL framework models the IRL problem using a Bayesian posterior: $P(R | O) \propto P(O | R)P(R)$, where $R$ denotes the reward function and $O$ represents the observed expert actions.
The likelihood model $P(O | R)$ assumes the expert is attempting to be optimal. The probability of an action is modeled using a Boltzmann distribution based on the optimal Q-values ($Q^*$) for a given reward $R$. This formulation allows the model to tolerate sub-optimal actions or "noise" from the expert. Furthermore, the framework supports various priors $P(R)$, such as Laplacian priors for sparse rewards or Ising models for spatial correlations in grid-like environments.
Sampling with PolicyWalk
To sample from the complex posterior distribution, the authors developed PolicyWalk, a specialized Markov Chain Monte Carlo (MCMC) algorithm. PolicyWalk efficiently explores the reward space by tracking the optimal policy. Since small perturbations in the reward function often do not change the optimal policy, the algorithm can use Policy Iteration to rapidly update Q-values rather than re-solving the MDP from scratch at each step.
A key theoretical contribution is the Mean Reward Theorem, which states that for apprenticeship learning, the optimal policy to adopt is the one that is optimal with respect to the mean of the posterior reward distribution. This provides a clear bridge between reward estimation and behavior cloning.
Experimental Results
The authors evaluated BIRL against standard heuristic-based IRL methods across various environments, including random MDPs and a grid-world "Adventure Game."
As shown in the experimental results, BIRL significantly outperforms previous methods in reward loss (the distance between estimated and true rewards). The Bayesian approach demonstrated superior scaling as the number of states ($N$) increased. Additionally, the use of informative priors, such as the Ising prior in the Adventure Game task, led to faster and more accurate learning by incorporating spatial domain knowledge.
Conclusion
Bayesian Inverse Reinforcement Learning provides a mathematically rigorous foundation for reward inference. By representing the reward as a distribution rather than a point estimate, the framework naturally handles the inherent uncertainty of the IRL problem. While scaling to continuous state spaces remains a challenge for future work, BIRL represents a significant step forward in apprenticeship learning and preference elicitation.
Summary updated