Abstract
Our goal is for agents to optimize the right reward function, despite how difficult it is for us to specify what that is. Inverse Reinforcement Learning (IRL) enables us to infer reward functions from demonstrations, but it usually assumes that the expert is noisily optimal. Real people, on the other hand, often have systematic biases: risk-aversion, myopia, etc. One option is to try to characterize these biases and account for them explicitly during learning. But in the era of deep learning, a natural suggestion researchers make is to avoid mathematical models of human behavior that are fraught with specific assumptions, and instead use a purely data-driven approach. We decided to put this to the test -- rather than relying on assumptions about which specific bias the demonstrator has when planning, we instead learn the demonstrator's planning algorithm that they use to generate demonstrations, as a differentiable planner. Our exploration yielded mixed findings: on the one hand, learning the planner can lead to better reward inference than relying on the wrong assumption; on the other hand, this benefit is dwarfed by the loss we incur by going from an exact to a differentiable planner. This suggest that at least for the foreseeable future, agents need a middle ground between the flexibility of data-driven methods and the useful bias of known human biases.
Introduction
To build AI systems that truly assist humans, agents must understand human preferences. Inverse Reinforcement Learning (IRL) is the standard framework for this, inferring reward functions from observed behavior. However, traditional IRL relies on the assumption of "Boltzmann rationality"—the idea that humans are near-optimal but occasionally make random errors. In reality, human behavior is riddled with systematic biases, such as myopia, overconfidence, and time-inconsistency. If an agent assumes a biased human is rational, it will fundamentally misinterpret their goals.
Proposed Method: Learning the Planner
Instead of hard-coding a specific (and likely incorrect) model of human bias, this paper investigates whether we can learn the human's internal planning algorithm directly from data. The authors propose treating the human decision-making process as a differentiable planner. They utilize a Value Iteration Network (VIN)—a neural network architecture that embeds a planning algorithm—to represent how a human maps rewards to actions.
They introduce two primary algorithmic approaches:
- IRL-with-rewards: Used when some tasks have known rewards. The VIN is trained to mimic human behavior on these tasks, then frozen to infer rewards in new environments.
- IRL-without-rewards: A more challenging setting where rewards are never known. The planner is initialized as optimal and then fine-tuned alongside reward inference to capture human-like deviations.
Experimental Setup
The approach was tested in 14x14 gridworld environments using synthetic demonstrators programmed with specific, known biases. These included Myopia (short-sightedness), Hyperbolic Discounting (time-inconsistency), and Underconfidence (misjudging costs or success probabilities).
Key Findings and the "Approximation Gap"
The research yielded several critical insights regarding the feasibility of this data-driven approach:
- The Impossibility Wall: It is theoretically impossible to learn both reward and bias from scratch without any grounding. The agent needs either known tasks or a strong prior (e.g., assuming the human is "close" to optimal) to distinguish between a strange reward and a strange bias.
- Performance Gains: Learning the planner outperformed baselines that assumed Boltzmann rationality. When demonstrators were biased, the proposed methods achieved ~89% of the maximum possible reward, compared to ~67% for the "Assume Optimal" baseline.
- The Bottleneck: A significant bottleneck was discovered: the errors introduced by using an approximate neural network planner (like a VIN) often outweighed the benefits of modeling the specific bias. When the authors replaced the neural VIN with an exact differentiable planner, performance rose to ~98%.
Conclusion
While learning human biases from data is a promising path forward, current neural planning architectures are a limiting factor. The results suggest that the "noise" in the AI's own planning ability currently limits its ability to model human bias accurately. Future agents will likely need a hybrid approach—combining the flexibility of data-driven learning with the structural benefits of known human behavioral models.
Summary updated