In the world of machine learning, we often try to "imitate the expert." We observe a doctor diagnosing patients or a pilot flying a plane, and we train a model to replicate their actions. However, there is a major flaw in this standard approach: it usually assumes the expert is a static, perfect machine. In reality, humans are constantly learning, adapting, and—crucially—making mistakes based on their past experiences.
This paper by Alex J. Chan and colleagues introduces Inverse Online Learning (IOL), a framework designed to look at a sequence of human decisions and work backward. The goal is to figure out not just what the experts are doing, but how their internal beliefs and decision-making processes are evolving over time.
The Problem: Moving Beyond Static Experts
Traditional imitation learning (IL) and inverse reinforcement learning (IRL) typically assume stationarity: the idea that the expert’s policy doesn't change. If a doctor chooses Treatment A today, these models assume they would choose it for the exact same patient tomorrow.
But humans are inherently "reactionary." We learn "online" from the consequences of our actions. If a clinician tries a new protocol and the patient reacts poorly, they might be less likely to use it next time, even if the bad outcome was just a statistical fluke. Standard models fail to capture this "learning-on-the-fly" behavior. The authors aim to bridge this gap by treating policy inference as the inverse of an online learning problem, allowing us to model non-stationary policies that evolve as the agent gains experience.
Methods: Modeling the "Perceived" Reality
The authors propose a novel architecture based on Deep State-Space Models (DSSMs) and the Potential Outcomes framework.
- The "Perceived Effect" Framework: Instead of assuming the agent knows the "true" effect of an action, the model assumes the agent has a perceived treatment effect. They choose actions they believe will lead to the best outcome, even if those beliefs are flawed.
- Memory Aggregation: A deep state-space model tracks a latent variable $M_t$ (memory). This memory acts as a "sufficient statistic" of everything the agent has seen so far, representing their current state of knowledge.
- Outcome Estimation: The model decodes this memory into weights for a linear model. At each time step, the agent's "mind" uses these weights and the current patient's data (context) to predict potential outcomes.
- The Inverse Algorithm: To train this, the authors use Variational Inference. They use an inference network (featuring a backwards LSTM) to look at a whole trajectory of decisions and outcomes to estimate the most likely path of the agent's internal belief updates.
Essentially, the model learns the "update rule" the human is using, even if that rule is biased or mathematically imperfect.
Key Insights and Ideas
By applying IOL to several high-stakes medical domains—including UNOS (liver transplant decisions), ICU antibiotic prescriptions, and Cystic Fibrosis treatments—the researchers uncovered several fascinating behavioral patterns:
- Beliefs vs. Reality: The model doesn't care if the agent is "right." It focuses on what the agent thinks is right. This allows it to capture cognitive biases, such as over-weighting recent failures.
- The MELD Score Shift: In the UNOS dataset, the model successfully captured a historical shift. When the MELD score (a measure of sickness) was introduced in 2002, it was the primary driver for organ acceptance. Over time, the model showed that clinicians began to "de-prioritize" it as they realized its limitations, leading to a more stabilized, multi-factor policy.
- Asymmetric Learning: The researchers found that agents are often more "reactionary" to negative outcomes than positive ones. For example, a rejected organ offer followed by a patient death shifted the decision boundary significantly more than a "success" story.
Key Results
- Better Action Matching: On the UNOS dataset, IOL achieved an AUC of 0.824, outperforming standard Behavioral Cloning (0.803) and Counterfactual IRL (0.553). This suggests that accounting for the change in policy is vital for accurately predicting what a human will do next.
- Robustness across Domains: In the ICU, Cystic Fibrosis, and Hospital Wards datasets, IOL consistently matched or beat state-of-the-art baselines in predicting agent actions, proving its versatility.
- Interpretability: Unlike "black-box" imitation models, IOL provides a trajectory of weights (e.g., how much the agent valued "Patient Age" vs. "Creatinine levels" at any specific month in 2005), offering a transparent window into the decision-maker's mind.
Limitations and Future Work
While powerful, Inverse Online Learning has its constraints. As a deep learning-based approach, it requires a significant amount of trajectory data to learn complex update functions. Furthermore, the model operates on a "Good Intent" assumption—it assumes agents are always trying to maximize outcomes. It cannot currently account for "exploratory" strategies (where an agent intentionally picks a sub-optimal action just to learn more) or malicious intent.
The authors also warn of the ethics of intent inference: because this is an "inverse" problem, the model provides a plausible explanation for behavior, not an absolute truth. There is a risk of misattributing incorrect intentions to human experts if the model is used carelessly. Future directions may include integrating a measure of "expected information gain" to handle agents who are actively experimenting.
Summary updated