Abstract
Decision analysis deals with modeling and enhancing decision processes. A principal challenge in improving behavior is in obtaining a transparent description of existing behavior in the first place. In this paper, we develop an expressive, unifying perspective on inverse decision modeling: a framework for learning parameterized representations of sequential decision behavior. First, we formalize the forward problem (as a normative standard), subsuming common classes of control behavior. Second, we use this to formalize the inverse problem (as a descriptive model), generalizing existing work on imitation/reward learning -- while opening up a much broader class of research problems in behavior representation. Finally, we instantiate this approach with an example (inverse bounded rational control), illustrating how this structure enables learning (interpretable) representations of (bounded) rationality -- while naturally capturing intuitive notions of suboptimal actions, biased beliefs, and imperfect knowledge of environments.
Introduction
In real-world scenarios, decision-makers—such as clinicians, economists, or autonomous systems—rarely exhibit "perfect rationality." Instead, they are boundedly rational, constrained by limited information-processing capacity, biased beliefs, and imperfect environmental knowledge. Standard imitation learning focuses on what an agent does, while Inverse Reinforcement Learning (IRL) focuses on what they want (rewards). However, these approaches often fail to explain why an agent behaves suboptimally. For instance, a clinician might make a mistake due to incorrect goals, over-cautiousness, or cognitive overload.
To address this, we introduce Inverse Decision Modeling (IDM), a framework that unifies these approaches to recover interpretable traits like optimism, flexibility, and adaptivity from observed demonstrations. By treating behavior as a projection onto a structured, parameterized space, IDM provides a descriptive analysis capable of quantifying human imperfections transparently.
The Inverse Decision Modeling Framework
The IDM framework operates by defining two primary components: the Forward Problem and the Inverse Problem.
- The Forward Problem ($F$): This defines how an agent, characterized by specific parameters $\theta$ (e.g., rewards, discount factors, or cognitive biases), generates behavior.
- The Inverse Problem ($G$): Given a set of observed behaviors, this component infers the parameters $\theta$ that most likely produced them.
Inverse Bounded Rational Control (IBRC)
We instantiate IDM through Inverse Bounded Rational Control (IBRC), which uses information-theoretic constraints to model three specific types of boundedness:
- Flexibility ($\alpha^{-1}$): Measures the consistency of "best" action selection versus stochastic behavior.
- Optimism ($\beta^{-1}$): Determines if the agent's internal model is biased toward positive or negative outcomes.
- Adaptivity ($\eta^{-1}$): Quantifies the agent's willingness to update beliefs based on novel information.
Technically, IBRC solves a Lagrangian optimization where the agent maximizes utility subject to complexity costs for information processing, resulting in generalized recursions for value functions and policies.
Experimental Evaluation
We evaluated the IDM framework using both synthetic and real-world datasets.
Simulated Environment: DIAG
The DIAG environment is a synthetic medical diagnostic POMDP used to verify the model's ability to recover "ground truth" bias parameters. In these simulations, the model successfully recovered the exact levels of flexibility, optimism, and adaptivity used to generate the data, demonstrating parameter identifiability.
Real-World Application: ADNI
Using data from the Alzheimer’s Disease Neuroimaging Initiative (ADNI), which tracks clinical visits for 1,626 patients, we analyzed clinician decisions regarding MRI scans. The IBRC model achieved high fidelity, matching the predictive performance of black-box RNN-based Behavioral Cloning (Calibration error of 0.16 vs. 0.18). Crucially, the model revealed that clinicians are significantly less optimistic (more cautious) when diagnosing patients carrying the ApoE4 genetic risk factor.
Discussion and Key Insights
IDM enables interpretability via projection, allowing researchers to describe behavior in human-understandable terms (e.g., "20% more pessimistic than ideal"). It excels at disentangling biases, distinguishing between different causes for the same observable action, such as stopping a test due to optimism versus a low adaptivity threshold.
While powerful, the framework faces challenges regarding identifiability; not all parameters can be inferred from behavior without specific structural assumptions. Future work will explore "attentionality" and methods for automatically discovering relevant parameter structures.
Summary updated