Sensitivity of Inverse Reinforcement Learning to Model Misspecification

21 concepts7 papers1 proposal

Inverse Reinforcement Learning (IRL) operates on a compelling premise: if we want to teach an AI a complex task or understand its underlying goals, we do not need to hand-craft a reward function. Instead, we can observe an agent (such as a human expert) and infer their rewards directly from their actions.

However, this inference process requires a bridge. To translate observed actions back into an unobserved reward function, an IRL algorithm must make two fundamental assumptions: a behavioral model (how the agent's internal rewards translate into choices) and an environmental model (how those choices interact with the physics and planning horizon of the world).

What happens when these assumptions are even slightly wrong? A growing body of research reveals that IRL is surprisingly fragile to this misspecification, while also providing the mathematical frameworks needed to understand and mitigate this sensitivity.

The Fragility of Behavioral Assumptions

Most classic IRL algorithms assume a clean, mathematically tractable relationship between rewards and actions. The simplest model assumes perfect optimality: the demonstrator always chooses the action that maximizes expected reward.

As demonstrated by Joar Max Viktor Skalse et al., this perfect optimality assumption exhibits exactly zero robustness to misspecification. If a human demonstrator makes even a minor mistake or deviates slightly from the absolute mathematical optimum, the entire reward inference framework breaks down, rendering the recovered reward function completely unreliable.

To cope with human suboptimality, practitioners frequently swap perfect optimality for probabilistic models, such as Boltzmann (softmax) rationality or the Bradley-Terry-Luce model for pairwise preferences. These models assume that humans choose suboptimal actions with a probability proportional to their expected value. Yet, as Joar Max Viktor Skalse et al. and Kihyun Kim et al. point out, human psychology rarely conforms to these rigid mathematical abstractions.

When actual human behavior deviates from these idealized probabilistic models, standard reward recovery methods fail to capture the true underlying goals, introducing significant bias or producing nonsensical reward functions. Furthermore, Daniel S. Brown et al. emphasize that relying blindly on softmax-rationality assumptions compromises the safety guarantees of downstream policies, especially when dealing with highly suboptimal or adversarial demonstrations.

Environmental Misspecification: The Hidden Trap

While it is intuitive that human psychology is hard to model, a more insidious vulnerability lies in how IRL models the environment itself. To infer why an agent took a specific path, the observer must know the environment's transition dynamics (its physics) and the agent's discount factor (its planning horizon).

In a pair of foundational papers, Joar Max Viktor Skalse et al. and Joar Max Viktor Skalse et al. rigorously quantified this sensitivity. They proved that even tiny errors in specifying environmental parameters—such as misestimating the discount rate by a fraction or slightly mismodeling the transition probabilities—propagate catastrophically through the reward inference process.

The resulting error can cause the inferred reward function to become completely orthogonal to the true reward function. In practice, this means a minor misestimate of the environment's physics can make the reward inference algorithm perform no better than a random guess.

[True Reward Function] ──(Small Environmental Error)──> [Orthogonal/Nonsensical Reward]

This environmental sensitivity is closely tied to how accurately we can estimate what the expert actually experienced. For instance, Abdeslam Boularias et al. highlighted that IRL is exceptionally sensitive to inaccuracies in calculating feature expectations from limited demonstrations. If the observer miscalculates how often an expert visits specific environmental features, the underlying reward function fails to recover correctly.

Building Robustness: When Does Reward Learning Work?

Given these theoretical vulnerabilities, why does reward learning work at all in practice? The answer lies in identifying specific structural properties that bound these errors, or designing algorithms that bypass rigid assumptions entirely.

1. Log-Concavity as a Buffer

The authors of Joey Hong et al. provided a crucial theoretical silver lining. While adversarial misspecification can cause catastrophic failures, reward inference remains stable in practice if the human behavioral model satisfies a log-concavity property. Under this condition, the error in the recovered reward function is bounded linearly by the error in the human model, preventing the orthogonal divergence proved in worst-case analyses.

2. Primal-Dual Optimization

Other researchers have focused on designing algorithms explicitly engineered to bypass rigid behavioral assumptions. For example, Kihyun Kim et al. developed a unified linear programming framework for offline reward learning that shifts away from traditional maximum likelihood estimation. By utilizing a primal-dual approach, their method achieves provable sample efficiency and robustness against irrational human behaviors that violate standard preference models.

3. Active Risk Mitigation

Alternatively, instead of passively accepting demonstration data, agents can actively manage risk. Daniel S. Brown et al. introduced ActiveVaR, an active learning algorithm that selects queries based on performance risk rather than simple reward uncertainty. By proactively querying the demonstrator in states where the current policy faces the highest potential for worst-case generalization error, the system bypasses the vulnerabilities of static behavioral assumptions and establishes a mathematically grounded stopping criterion for safe deployment.

Key Foundational Work

Bootstrapping Apprenticeship Learning (2010) — contributed 1 concept elements to this direction.
Risk-Aware Active Inverse Reinforcement Learning (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_behavioral_model_misspecification_cf94f3d1,
  title = {behavioral model misspecification},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/behavioral-model-misspecification/rdr_cf94f3d14c614635a597a468b62ec7e6},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_behavioral_model_misspecification_cf94f3d1_v1,
  title = {behavioral model misspecification},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/behavioral-model-misspecification/rdr_cf94f3d14c614635a597a468b62ec7e6/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake