Abstract
Existing observational approaches for learning human preferences, such as inverse reinforcement learning, usually make strong assumptions about the observability of the human's environment. However, in reality, people make many important decisions under uncertainty. To better understand preference learning in these cases, we study the setting of inverse decision theory (IDT), a previously proposed framework where a human is observed making non-sequential binary decisions under uncertainty. In IDT, the human's preferences are conveyed through their loss function, which expresses a tradeoff between different types of mistakes. We give the first statistical analysis of IDT, providing conditions necessary to identify these preferences and characterizing the sample complexity -- the number of decisions that must be observed to learn the tradeoff the human is making to a desired precision. Interestingly, we show that it is actually easier to identify preferences when the decision problem is more uncertain. Furthermore, uncertain decision problems allow us to relax the unrealistic assumption that the human is an optimal decision maker but still identify their exact preferences; we give sample complexities in this suboptimal case as well. Our analysis contradicts the intuition that partial observability should make preference learning more difficult. It also provides a first step towards understanding and improving preference learning methods for uncertain and suboptimal humans.
Introduction
How do we learn what a human wants just by watching them? This is the core challenge of Inverse Reinforcement Learning (IRL). However, IRL faces major hurdles: underspecification, where many reward functions explain the same behavior, and the often unrealistic assumption of human optimality. Contrary to the intuition that partial information or uncertainty makes modeling behavior harder, this work proves that uncertainty actually makes it easier to learn human preferences. By analyzing Inverse Decision Theory (IDT), we show that "clear-cut" decisions reveal almost nothing about a person's internal trade-offs, whereas difficult, uncertain choices allow us to mathematically recover their exact loss functions.
Inverse Decision Theory and Methodology
Unlike standard IRL, which often looks at complex sequences of actions, IDT focuses on a simpler setting: a human making a single binary choice (e.g., "to treat or not to treat") based on uncertain information. The human's preferences are represented by a "cost matrix"—the internal weights they give to different types of mistakes, such as False Positives versus False Negatives.
The methodology involves a rigorous statistical analysis of IDT to provide bounds on sample complexity. We move beyond the "optimal human" assumption by analyzing cases of known and unknown suboptimality. In the case of known suboptimality, the human is assumed to choose the best rule from a limited, known hypothesis class of decision rules.
The Informativeness of Uncertainty
A key insight is the "informativeness" of hard choices. If a decision is obvious (e.g., a test is 100% accurate), every person—regardless of their preferences—will make the same choice, revealing nothing about their values. However, if the test is only 60% accurate, the choice to "risk it" or "play it safe" reveals the internal trade-off between different types of errors.
Uncertainty ensures that the probability density around a decision threshold is non-zero. This "spread" of data points allows an observer to see exactly where the decision-maker "switches" from one choice to another. Even if a human is "suboptimal" (e.g., they ignore certain data features), as long as their behavior is consistent within a certain type of strategy, their underlying preferences can still be recovered.
Key Results
- Sample Complexity: For an optimal agent, the number of observations $m$ needed to learn a preference to precision $\epsilon$ is $\Theta(1/(p_c \epsilon))$. Crucially, as uncertainty increases (higher probability density $p_c$ at the threshold), the number of samples needed decreases.
- The Impossibility of Certainty: If there is no uncertainty in the environment, the loss function is unidentifiable. Trade-offs cannot be learned from "obvious" decisions.
- Suboptimal Learning: For humans using unknown restricted strategies, the sample complexity is higher—growing with $1/\epsilon^2$ and the VC-dimension of the possible strategies—but preferences remain identifiable.
- Fairness Auditing: IDT can be used to test for group calibration, detecting if a decision-maker applies different "loss parameters" (biases) to different demographic groups.
Discussion and Future Work
While this work establishes statistical limits, it assumes the observer knows the true distribution of the data. Future research should address estimating this distribution from samples. Additionally, extending these insights to sequential decision-making (POMDPs) and integrating psychological models like Prospect Theory, which accounts for non-linear perceptions of risk, are vital next steps. Finally, the computational efficiency of these inverse learning algorithms in high-dimensional spaces remains an open area for investigation.
Summary updated