Explaining by Imitating: Understanding Decisions by Interpretable Policy Learning

Alihan Hüyük, Daniel Jarrett, Cem Tekin, Mihaela van der Schaar

In high-stakes fields like healthcare, understanding why a professional makes a specific decision is often as important as the decision itself. However, most machine learning models used to mimic human behavior are "black boxes"—they might predict an action accurately, but they cannot explain the reasoning behind it.

To address this, researchers from the University of Cambridge and Bilkent University developed INTERPOLE (Interpretable Policy Learning), a framework designed to "explain by imitating." By decomposing human behavior into transparent belief-update processes and clear decision boundaries, INTERPOLE offers a way to audit, quantify, and truly understand the nuances of expert decision-making.

The Challenge of Modeling Human Behavior

Modeling human behavior in the real world is notoriously difficult, primarily due to three constraints:

  1. Partial Observability: Decision-makers rarely have a perfect view of the "true state" of the world. For example, a doctor doesn't know for certain if a patient has early-stage Alzheimer's; they only infer it from test results and observations.
  2. Lack of Transparency: Standard imitation learning (like Behavioral Cloning) often uses Recurrent Neural Networks (RNNs). While highly accurate, an RNN's hidden states are uninterpretable, offering no insight into the decision-making rationale.
  3. Offline Constraints: In fields like medicine, we cannot perform "live" experiments to see how a doctor reacts to different scenarios. Models must learn entirely from historical, observational data.

Existing methods like Inverse Reinforcement Learning (IRL) attempt to find the "rewards" an agent is seeking, but they often assume the agent is acting optimally and has an unbiased view of the world—assumptions that rarely hold true for fallible humans.

The INTERPOLE Framework

INTERPOLE is a model-based Bayesian framework that learns to imitate human behavior while remaining "transparent by design." It models an agent's policy through two distinct, interpretable components:

  • Decision Dynamics: Instead of a black-box RNN, INTERPOLE uses an Input-Output Hidden Markov Model (IOHMM). This represents the agent’s subjective belief-update process. It tracks how the agent's internal "confidence" (or belief) shifts as they receive new observations over time.
  • Decision Boundaries: The model maps these internal beliefs to actions using "mean vectors" in a belief simplex. Imagine a triangle where each corner is a possible underlying state (e.g., Healthy, Mild Cognitive Impairment, Dementia). INTERPOLE identifies regions within that triangle where a specific action (like ordering an MRI) is most likely. The lines separating these regions are the "decision boundaries."
INTERPOLE models decision-making by tracking belief updates (left) and mapping them to actions via decision boundaries within a belief simplex (right).

Crucially, the Bayesian Expectation-Maximization (EM) learning algorithm used by INTERPOLE does not assume the agent's beliefs are "correct" or their actions are "optimal." It simply seeks the most plausible internal explanation for the observed behavior.

Experimental Validation and Key Results

The researchers validated INTERPOLE using both simulated environments (to verify ground-truth recovery and bias detection) and real-world data from the Alzheimer’s Disease Neuroimaging Initiative (ADNI), comprising 1,737 patients. The model was tasked with understanding when and why clinicians chose to order expensive MRI tests based on cognitive scores and patient history.

High Fidelity and Superior Explanation

In the ADNI dataset, INTERPOLE matched or outperformed black-box RNNs in predicting clinician actions, proving that interpretability does not require sacrificing accuracy. Furthermore, in simulated "biased" environments, INTERPOLE's error in recovering the true policy was orders of magnitude lower than benchmark algorithms, as it correctly identified when an agent was acting on a "false" belief.

Auditing Power and Visualizing the "Mind's Eye"

By plotting belief trajectories on a simplex, researchers can visually track a clinician's reasoning. You can see exactly when a new piece of data pushes a clinician across a boundary to take action.

Belief trajectories for different patient scenarios, illustrating how INTERPOLE can audit clinical decisions and identify anomalies like late diagnoses.

Using this auditing capability on the ADNI database, the authors identified that 6.5% of patients were "belatedly diagnosed"—meaning the model's learned "typical" policy suggested they should have been diagnosed months earlier. They also found that 19% of MRIs ordered were likely unnecessary, as they didn't significantly shift the clinician's internal beliefs.

When surveyed, clinicians unanimously preferred INTERPOLE's "belief trajectories" over raw data for understanding patient cases, and the majority preferred "decision boundaries" over "reward functions" for describing medical policies.

Limitations and Future Work

While powerful, INTERPOLE has a few limitations:

  • State Space Definition: The framework currently requires a predefined state space (e.g., the three stages of dementia). It may struggle in domains where the "states" are unknown or highly abstract.
  • Discrete Time: The current model operates in discrete time steps. Extending this to continuous-time settings would better model the irregular intervals between real-world hospital visits.
  • Scaling: While highly effective for the 3-state Alzheimer's problem, scaling the visualization and computation to much larger state spaces remains a challenge for future research.

Summary updated

Abstract

Understanding human behavior from observed data is critical for transparency and accountability in decision-making. Consider real-world settings such as healthcare, in which modeling a decision-maker’s policy is challenging—with no access to underlying states, no knowledge of environment dynamics, and no allowance for live experimentation. We desire learning a data-driven representation of decision- making behavior that (1) inheres transparency by design, (2) accommodates partial observability, and (3) operates completely offline. To satisfy these key criteria, we propose a novel model-based Bayesian method for interpretable policy learning (“Interpole”) that jointly estimates an agent’s (possibly biased) belief-update process together with their (possibly suboptimal) belief-action mapping. Through experiments on both simulated and real-world data for the problem of Alzheimer’s disease diagnosis, we illustrate the potential of our approach as an investigative device for auditing, quantifying, and understanding human decision-making behavior.
BibTeX
@inproceedings{huyuk2021explaining,
  author = {Alihan Hüyük and Daniel Jarrett and Cem Tekin and Mihaela van der Schaar},
  title = {{Explaining by Imitating: Understanding Decisions by Interpretable Policy Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=unI5ucw_Jk},
}
Figure