Inverse Reinforcement Learning (IRL) is often described as a detective story. We observe an expert’s actions—the "what"—and attempt to infer their underlying reward function—the "why." However, this process is fundamentally ill-posed. As Prashant Doshi and Saurabh Arora note in A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress, an infinite number of reward functions can explain the same observed behavior.
If a robot avoids a puddle, is it because it fears water, or because it simply prefers the traction of the pavement? Without resolving this ambiguity, the robot might fail catastrophically when the environment changes—for instance, by driving off a cliff just to stay on the pavement.
The Geometry of Ambiguity
The core of the "identifiability" problem lies in the relationship between policies and rewards. A single optimal policy can be rationalized by many different reward structures. Haoyang Cao, Samuel N. Cohen, and Lukasz Szpruch explain in Identifiability in inverse reinforcement learning that this relationship is mediated by a latent value function. Because any arbitrary function could potentially serve as the value function, the reward remains mathematically underdetermined.
This creates a "brittleness" in learned rewards. As Justin Fu and Sergey Levine demonstrate in Learning Robust Rewards with Adversarial Inverse Reinforcement Learning, rewards often become "entangled" with the specific physics of the training environment. To extract true intent, we must find ways to "disentangle" the reward from the environment's dynamics.
Triangulating Intent through Variation
One of the most powerful ways to resolve ambiguity is to observe the agent in multiple contexts. If we only see a driver in the sun, we don't know if they like the light or hate the rain. By observing them in both conditions, we can triangulate their true preferences.
Haoyang Cao and colleagues proved that observing an agent in just two different environments (or under two different discount rates) can uniquely identify the reward function up to a constant. Similarly, Alberto Maria Metelli and Marcello Restelli introduced TRAVEL in Provably Efficient Learning of Transferable Rewards, an algorithm that strategically queries an environment to ensure the recovered reward generalizes across different domains.
Strategic Teaching and Informative Failures
If passive observation is insufficient, we can turn to Machine Teaching. Instead of providing random demonstrations, a teacher can select the most "informative" examples that specifically distinguish between competing reward hypotheses. Daniel S. Brown and Scott Niekum developed SCOT (Set Cover Optimal Teaching) in Machine Teaching for Inverse Reinforcement Learning to identify the absolute minimum set of demonstrations needed to resolve this ambiguity.
Interestingly, what an agent cannot do is often as revealing as what it can do. Nyal Patel and Sonali Parbhoo proposed Failure-Aware IRL (FA-IRL) in Learning from Failures. By treating model failures as primary constraints, they narrow the set of feasible reward functions, leading to more stable reconstructions of internal preferences in complex models like LLMs.
The Challenge of Partial Observability
The problem of ambiguity scales with complexity. In multi-agent settings, observing a single equilibrium (like a Nash Equilibrium) is rarely enough to recover unique rewards, as many contradictory incentives can lead to the same strategic standoff (Giorgia Ramponi and Till Freihaut, On Multi-Agent Inverse Reinforcement Learning).
Perhaps most dangerously, Leon Lang and Stuart Russell have shown in When Your AIs Deceive You that partial observability creates irreducible ambiguity. When a human evaluator cannot see everything an AI is doing, multiple reward functions—some safe, some deceptive—can explain the same feedback. This suggests that resolving ambiguity is not just a mathematical hurdle for efficiency, but a fundamental requirement for AI safety.