Abstract
Understanding human behavior from observed data is critical for transparency and accountability in decision-making. Consider real-world settings such as healthcare, in which modeling a decision-maker's policy is challenging -- with no access to underlying states, no knowledge of environment dynamics, and no allowance for live experimentation. We desire learning a data-driven representation of decision-making behavior that (1) inheres transparency by design, (2) accommodates partial observability, and (3) operates completely offline. To satisfy these key criteria, we propose a novel model-based Bayesian method for interpretable policy learning ("Interpole") that jointly estimates an agent's (possibly biased) belief-update process together with their (possibly suboptimal) belief-action mapping. Through experiments on both simulated and real-world data for the problem of Alzheimer's disease diagnosis, we illustrate the potential of our approach as an investigative device for auditing, quantifying, and understanding human decision-making behavior.
The Challenge of Interpretable Policy Learning
In high-stakes domains like healthcare, understanding why a decision was made is as important as the decision itself. However, modeling human behavior is notoriously difficult. Decision-makers often operate under partial observability, forming subjective beliefs based on noisy observations rather than the true state of the world. Furthermore, human agents are rarely perfectly optimal; they may harbor biases or use flawed internal models of environment dynamics.
Existing approaches often fall short of providing a complete picture. Standard Imitation Learning (IL) methods like Behavioral Cloning (BC) or GAIL typically rely on "black-box" neural networks that lack transparency. Conversely, Apprenticeship Learning via Inverse Reinforcement Learning (IRL) focuses on high-level reward functions, which may fail to explain specific, action-level choices in complex, partially observable environments.
Methodology: INTERPOLE
To address these gaps, we introduce INTERPOLE (Interpretable Policy Learning). Our framework decomposes decision-making into two distinct, learned components:
- Decision Dynamics: Modeled as an Input-Output Hidden Markov Model (IOHMM), this component represents how an agent updates their internal beliefs ($b_t$) over time. Crucially, INTERPOLE learns the agent's subjective view of the world, allowing it to capture potential biases in how observations are processed.
- Decision Boundaries: The policy is parameterized by "mean vectors" in the belief space. An action is selected based on which vector is closest to the agent's current belief, effectively partitioning the belief simplex into clear, Voronoi-like "decision regions."
We employ a Bayesian Expectation-Maximization (EM) algorithm for training. The "E-step" estimates latent belief trajectories using a forward-backward procedure, while the "M-step" uses gradient ascent to maximize the posterior probability of observed actions, ensuring the model remains grounded in the agent's actual behavior.
Key Insights and Real-World Application
INTERPOLE's primary strength lies in its ability to visualize the "mind's eye" of the decision-maker. By mapping beliefs onto a simplex, we can observe behavior as a trajectory moving through different diagnostic regions.
In our evaluation using the Alzheimer’s Disease Neuroimaging Initiative (ADNI) dataset, INTERPOLE demonstrated several key advantages:
- Accuracy without Opacity: INTERPOLE achieved a Brier Score of 0.17, outperforming black-box RNN-based models (0.18) and IRL benchmarks (0.23–0.24). This indicates that transparency does not require a sacrifice in predictive performance.
- Bias Recovery: In simulated environments with known false-negative biases, INTERPOLE successfully recovered the agent's subjective decision boundaries where standard model-based methods failed.
- Auditing Utility: The model identified that 6.5% of patients in the ADNI cohort received "belated diagnoses"—cases where the learned "typical" clinical policy would have suggested a diagnosis significantly earlier than recorded.
Conclusion
INTERPOLE provides a robust framework for auditing and understanding human decision-making in complex, offline environments. By jointly learning subjective dynamics and transparent decision boundaries, it offers a path toward more accountable AI in clinical settings. Future work will explore extending this framework to continuous state spaces and real-time decision support.
Summary updated