Inverse Reinforcement Learning by Estimating Expertise of Demonstrators

Mark Beliaev, Ramtin PedarsaniAAAI 2025 · DOI

TL;DR

This paper introduces IRLEED, a novel Inverse Reinforcement Learning (IRL) framework that tackles the challenge of learning from suboptimal and heterogeneous expert demonstrations. By modeling individual demonstrator expertise (reward bias and action variance), IRLEED significantly outperforms standard IRL methods, leading to more accurate reward recovery and better learned policies.

What Problem Does This Paper Address?

Imitation Learning (IL) aims to teach agents desired behaviors by observing expert demonstrations. A common approach is Inverse Reinforcement Learning (IRL), which infers the underlying reward function that explains the expert's actions. However, real-world demonstrations are often imperfect: they can be suboptimal (not truly expert-level) and heterogeneous (coming from different "experts" with varying skill levels and biases). Standard IRL algorithms typically assume all demonstrations are from a single, optimal expert, leading to poor performance when faced with noisy or varied data. Previous attempts to address this often relied on impractical assumptions like requiring a subset of truly expert data or explicit knowledge of the environment. This paper addresses the critical gap of learning effectively from such imperfect, mixed-quality demonstrations without prior knowledge of their quality.

Methods

The core innovation of IRLEED lies in its two-pronged approach:

  1. Suboptimal Demonstrator Model: IRLEED employs a generalized Boltzmann rationality model. This model captures two key aspects of a demonstrator's suboptimality:
  2. Accuracy ($\epsilon_i$): This represents a "reward bias," where the demonstrator's perceived reward deviates from the true reward.
  3. Precision ($\beta_i$): This represents "action variance," controlling how much the demonstrator's actions deviate from the optimal choice given their perceived reward. This model allows IRLEED to individually characterize each demonstrator's expertise level.

  4. Maximum Entropy IRL Framework: IRLEED integrates this demonstrator model into a Maximum Entropy IRL framework. Instead of assuming a single expert policy, it jointly estimates the true reward function ($\theta$) and the suboptimality parameters ($\epsilon_i, \beta_i$) for each demonstrator. This is achieved by maximizing a joint likelihood function over all demonstrations. The learning process iteratively updates the reward function, the suboptimality parameters, and the demonstrator policies.

Datasets

The paper evaluates IRLEED on several datasets:

  • Simulated Gridworld: A custom environment used to systematically test IRLEED's performance under varying levels of demonstrator accuracy and precision.
  • Simulated Control Tasks: Cartpole and Lunar Lander environments, where suboptimal demonstrations were generated.
  • Atari Environments: Space Invaders and Qbert, using a combination of simulated and human-generated demonstrations.
  • Mujoco Task: Hopper, using demonstrations from pretrained policies of varying expertise.

The paper also mentions using the ILEED (Beliaev et al., 2022) and IQ (Garg et al., 2021) codebases for comparison, implying these are standard benchmarks for such tasks.

Key Insights and Ideas

  • Suboptimality is Multi-faceted: The paper highlights that demonstrator imperfection isn't just about random noise (variance) but also about systematic biases in their perceived rewards. Modeling both accuracy ($\epsilon$) and precision ($\beta$) is crucial.
  • Heterogeneity is Key: Treating all demonstrations as coming from a single, albeit suboptimal, source (as standard IRL might do) leads to an "averaged" policy that is worse than learning from individual expertise. IRLEED's ability to differentiate demonstrators is its strength.
  • IRLEED Generalizes: The framework naturally recovers standard IRL when demonstrations are optimal ($\epsilon_i=0$) and precision is high ($\beta_i=1$). It also recovers the ILEED framework in a simplified, dynamics-agnostic setting.
  • Better Reward Recovery: By accounting for demonstrator biases, IRLEED can recover a more accurate representation of the true reward function, even when demonstrations are significantly misaligned with it.

Key Results

  • Gridworld: IRLEED showed a 30.3% improvement in relative performance over standard IRL on average, particularly excelling when demonstrator accuracy was low and precision was high.
  • Simulated Control Tasks (Cartpole, Lunar Lander): In online settings, IRLEED consistently outperformed its ablated counterpart (IQ) and ILEED, especially in the more complex Lunar Lander environment, often performing on par or better than the best demonstrator.
  • Atari Environments: IRLEED showed a 17% improvement over IQ and a 41% improvement over ILEED when trained on a combined dataset of simulated and human demonstrations, demonstrating its robustness to varied data sources.
  • Mujoco Task (Hopper): IRLEED showed improved performance over IQ when trained on a mixture of demonstrations, filtering out suboptimal ones.

Limitations and Future Work

  • Reward Recovery Interpretation: While IRLEED aims to recover the true reward, interpreting this recovered reward can be challenging in complex, continuous environments. The paper primarily focuses on policy performance rather than reward recovery metrics in these cases.
  • Model Complexity: Simultaneously learning multiple demonstrator models adds complexity to the framework.
  • Uniqueness of Solution: The true reward parameter ($\theta^*$) is not guaranteed to be a unique maximizer of the IRLEED objective, meaning IRLEED might not always recover the exact true reward parameter, though it aims to recover a policy that performs optimally.
  • Future Work: The authors suggest further analysis of design choices like the demonstrator model and reward regularization. They also note that theoretical guarantees for reward recovery compared to IRL would be a valuable future direction, though they acknowledge this might require additional assumptions. The paper also hints at extending IRLEED with generalized IRL for complex function approximation and exploring its use in offline settings with techniques like Inverse Soft-Q learning.

Summary updated

BibTeX
@inproceedings{beliaev2025inverse,
  author = {Mark Beliaev and Ramtin Pedarsani},
  title = {{Inverse Reinforcement Learning by Estimating Expertise of Demonstrators}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i15.33705},
}
Figure