Inverse Online Learning: Understanding Non-Stationary and Reactionary Policies

Alex J. Chan, Alicia Curth, Mihaela van der Schaar

Abstract

Traditional imitation learning and inverse reinforcement learning frameworks typically assume that an agent’s policy is stationary—that the decision-maker has already converged on an optimal strategy. However, in complex, real-world domains such as medicine and finance, agents are constantly learning and adapting their strategies based on observed outcomes. This paper introduces Inverse Online Learning (IOL), a framework designed to model these non-stationary and reactionary policies. By combining deep state-space models with a potential outcomes framework, IOL reconstructs an agent's evolving internal beliefs from their observed actions, providing a more accurate and interpretable model of human decision-making over time.

Introduction

Most existing models for policy estimation assume that the underlying decision-making rule is fixed. While this simplifies the mathematics of imitation learning, it fails to capture the dynamic nature of human expertise. Experts are not static; they react to failures, integrate new information, and shift their priorities as they gain experience. For instance, a clinician’s willingness to perform a risky procedure may decrease immediately following a complication, or a trader may adjust their risk tolerance based on recent market volatility.

To address this, the authors propose Inverse Online Learning (IOL). Unlike standard Behavioral Cloning (BC), which maps states directly to actions, IOL seeks to understand the learning process itself. It treats the observed sequence of actions as the output of an online learning algorithm, allowing researchers to infer how an agent’s internal "worldview" changes in response to past experiences.

Methodology

The IOL framework models the agent as an online learner who updates their beliefs recursively. The architecture is composed of three primary modules:

1. Memory Aggregation ($S_\theta$)

To capture the non-stationary nature of the policy, the model uses a deep state-space model to summarize the agent's history. This module aggregates past contexts, actions, and outcomes into a latent memory variable, $M_t$. This memory serves as a proxy for the agent's current internal state or "experience level," evolving as new data points are processed.

2. Outcome Estimation ($O_\theta$)

The model decodes the latent memory $M_t$ into a set of weights. These weights are applied to a linear model to predict the agent’s perceived outcome of a given action. A critical distinction in IOL is that it does not attempt to model the objective ground truth of the environment; instead, it models the agent's subjective perception. This allows the framework to account for human biases, such as the tendency to over-weight recent events or ignore low-probability risks.

3. Treatment Rule ($\pi_\theta$)

The agent is assumed to follow a principle of "Perceived Optimality." They choose the action they believe will yield the highest expected utility based on their current (and potentially biased) outcome estimation. To make this process end-to-end trainable, the authors utilize a differentiable "soft" version of this selection rule.

Inference via Variational Methods

Training the IOL model requires inferring the hidden sequence of belief states that best explains the observed behavior. The authors employ Variational Inference, utilizing a "backwards LSTM" that processes the entire trajectory of an agent’s decisions in reverse. This allows the model to look at the totality of an agent's life to determine the most likely evolution of their internal decision-making parameters.

Experimental Evaluation

Synthetic Validation

The authors first validated IOL on synthetic datasets where the ground-truth belief updates were known. The model successfully recovered the underlying non-stationary policies and the specific mechanisms by which the synthetic agents updated their beliefs after receiving feedback.

UNOS Liver Transplant Case Study

The framework was applied to real-world data from the United Network for Organ Sharing (UNOS), specifically focusing on liver transplant acceptance decisions between 2003 and 2007. This period is particularly interesting as it followed the introduction of the MELD (Model for End-Stage Liver Disease) score, a standardized metric for prioritizing patients.

IOL was tasked with predicting whether a clinician would accept or decline a specific organ for a specific patient. The results demonstrated that IOL significantly outperformed traditional benchmarks:

  • IOL AUC: 0.824
  • Deep Behavioral Cloning (BC): 0.803
  • Linear Behavioral Cloning: 0.798

Key Insights

Interpretability and Clinical Evolution

Because the outcome estimation layer is locally linear, IOL provides high interpretability. By examining the weights assigned to different patient features over time, the authors could visualize the evolution of clinical priorities. For example, the model captured a declining reliance on the MELD score. While MELD was the primary driver of decisions shortly after its introduction, IOL showed that clinicians gradually de-emphasized it in favor of a more holistic, non-stationary approach as they learned the score's practical limitations through experience.

Reactionary Shifts

The model effectively captured "reactionary" behavior—sudden shifts in policy triggered by specific outcomes. If a clinician experienced a negative outcome after a certain type of transplant, the IOL model showed a measurable shift in the decision boundary for the subsequent time step, reflecting a temporary increase in caution that a stationary model would simply treat as noise.

Limitations and Future Work

While powerful, IOL is data-intensive, requiring longitudinal trajectories to accurately map latent belief evolution. Furthermore, the current framework assumes agents are always attempting to maximize outcomes. It does not yet account for "exploratory" behavior, where an agent might intentionally take a sub-optimal action to gather information. Future research may extend IOL to include "Value of Information" metrics, allowing the model to distinguish between genuine errors and purposeful experimentation.

Conclusion

Inverse Online Learning represents a significant step forward in imitation learning by moving beyond the assumption of static expertise. By modeling the agent as a dynamic learner, IOL provides a more nuanced and accurate reconstruction of human decision-making, offering both superior predictive performance and valuable insights into how professional expertise evolves in high-stakes environments.

Summary updated

BibTeX
@inproceedings{chan2022inverse,
  author = {Alex J. Chan and Alicia Curth and Mihaela van der Schaar},
  title = {{Inverse Online Learning: Understanding Non-Stationary and Reactionary Policies}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=DYypjaRdph2},
}
Figure