Resolving Reward Ambiguity in Inverse Reinforcement Learning

You are reading version 1 of this direction, frozen 2026-10-04. The latest version is version 2.

Version 1 of 2337 papers482 conceptsfrozen 2026-10-04

Inverse Reinforcement Learning (IRL) is often described as a detective story. We observe an expert’s actions—the "what"—and attempt to infer their underlying reward function—the "why." However, this process is fundamentally ill-posed. As Prashant Doshi and Saurabh Arora note in A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress, an infinite number of reward functions can explain the same observed behavior.

If a robot avoids a puddle, is it because it fears water, or because it simply prefers the traction of the pavement? Without resolving this ambiguity, the robot might fail catastrophically when the environment changes—for instance, by driving off a cliff just to stay on the pavement.

The Geometry of Ambiguity

The core of the "identifiability" problem lies in the relationship between policies and rewards. A single optimal policy can be rationalized by many different reward structures. Haoyang Cao, Samuel N. Cohen, and Lukasz Szpruch explain in Identifiability in inverse reinforcement learning that this relationship is mediated by a latent value function. Because any arbitrary function could potentially serve as the value function, the reward remains mathematically underdetermined.

This creates a "brittleness" in learned rewards. As Justin Fu and Sergey Levine demonstrate in Learning Robust Rewards with Adversarial Inverse Reinforcement Learning, rewards often become "entangled" with the specific physics of the training environment. To extract true intent, we must find ways to "disentangle" the reward from the environment's dynamics.

Triangulating Intent through Variation

One of the most powerful ways to resolve ambiguity is to observe the agent in multiple contexts. If we only see a driver in the sun, we don't know if they like the light or hate the rain. By observing them in both conditions, we can triangulate their true preferences.

Haoyang Cao and colleagues proved that observing an agent in just two different environments (or under two different discount rates) can uniquely identify the reward function up to a constant. Similarly, Alberto Maria Metelli and Marcello Restelli introduced TRAVEL in Provably Efficient Learning of Transferable Rewards, an algorithm that strategically queries an environment to ensure the recovered reward generalizes across different domains.

Strategic Teaching and Informative Failures

If passive observation is insufficient, we can turn to Machine Teaching. Instead of providing random demonstrations, a teacher can select the most "informative" examples that specifically distinguish between competing reward hypotheses. Daniel S. Brown and Scott Niekum developed SCOT (Set Cover Optimal Teaching) in Machine Teaching for Inverse Reinforcement Learning to identify the absolute minimum set of demonstrations needed to resolve this ambiguity.

Interestingly, what an agent cannot do is often as revealing as what it can do. Nyal Patel and Sonali Parbhoo proposed Failure-Aware IRL (FA-IRL) in Learning from Failures. By treating model failures as primary constraints, they narrow the set of feasible reward functions, leading to more stable reconstructions of internal preferences in complex models like LLMs.

The Challenge of Partial Observability

The problem of ambiguity scales with complexity. In multi-agent settings, observing a single equilibrium (like a Nash Equilibrium) is rarely enough to recover unique rewards, as many contradictory incentives can lead to the same strategic standoff (Giorgia Ramponi and Till Freihaut, On Multi-Agent Inverse Reinforcement Learning).

Perhaps most dangerously, Leon Lang and Stuart Russell have shown in When Your AIs Deceive You that partial observability creates irreducible ambiguity. When a human evaluator cannot see everything an AI is doing, multiple reward functions—some safe, some deceptive—can explain the same feedback. This suggests that resolving ambiguity is not just a mathematical hurdle for efficiency, but a fundamental requirement for AI safety.

Papers in this version

Showing 60 of 337 papers.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 308 papers
  2. Version 1 · 2026-10-04 · 337 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_reward_function_ambiguity_35529a5f,
  title = {reward function ambiguity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/reward-function-ambiguity/rdr_35529a5f5db34cad92d4baa7e324c7b1},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_resolving_reward_ambiguity_in_inverse_re_35529a5f_v1,
  title = {Resolving Reward Ambiguity in Inverse Reinforcement Learning},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/resolving-reward-ambiguity-in-inverse-reinforcement-learning/rdr_35529a5f5db34cad92d4baa7e324c7b1/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}