Treatment Policy Learning in Multiobjective Settings with Fully Observed Outcomes

Sooraj Boominathan, Michael Oberst, Helen Zhou, Sanjat Kanjilal, David A. Sontag

This paper, presented at KDD 2020, tackles a common but tricky scenario in medical AI: learning how to treat patients when you have multiple competing goals and a unique kind of "perfect" historical data. Using antibiotic prescription for Urinary Tract Infections (UTIs) as a case study, the authors demonstrate how machine learning can significantly outperform human clinicians by navigating the trade-offs between individual patient health and broader public health concerns.

TL;DR

The authors develop and compare methods for learning treatment policies in settings where historical data reveals the outcomes of all possible treatments, even though these outcomes are unknown at the moment a doctor must act. Their "direct" policy learning approach not only outperforms clinicians but also allows for seamless integration of complex goals, such as knowing when to defer a difficult case back to a human expert.

What Problem Does This Paper Address?

In many medical scenarios, doctors must make an "empiric" decision—prescribing a treatment immediately before definitive test results are available. For UTIs, a lab test can eventually show exactly which antibiotics would have worked for a specific patient. While this information arrives too late for the initial prescription, it creates a goldmine of historical data where we "fully observe" all potential outcomes.

The challenge is two-fold:

  1. Multi-objective Trade-offs: Doctors aren't just trying to cure the current patient (minimizing Inappropriate Antibiotic Therapy, or IAT). They also need to be "antibiotic stewards," avoiding the overuse of broad-spectrum (2nd-line) drugs that drive long-term population-level resistance.
  2. Model Misspecification: Traditional "indirect" methods try to predict the outcome of every drug and then pick the best one. However, if the underlying biology is complex but the optimal decision rule is simple, these predictive models might fail to find the best policy.

Methods

The researchers compare three distinct strategies:

  • Thresholding (Indirect): Train models to predict the probability of success for each drug. Use a set of thresholds to decide which drugs are "likely effective," then pick the cheapest/narrowest-spectrum option among them.
  • Expected Reward Maximization (Indirect): Combine effectiveness and cost into a single "reward" score using a weight ($\omega$) to represent the trade-off. Train models to predict this reward for each drug and pick the maximum.
  • Direct Policy Optimization: Instead of predicting outcomes, this method learns the decision rule directly. It optimizes a surrogate loss (multinomial deviance) to find a policy that maximizes the expected reward across the population.

A key advantage of the Direct approach is its ability to handle Learning to Defer. The authors simply add a "defer to doctor" action to the model's choices, incentivizing the model to step aside when it predicts the human clinician is likely to make a better (or equally good) decision.

Datasets

  • UTI Dataset: A real-world dataset of 15,806 specimens from two major Boston hospitals (MGH and BWH) collected between 2007 and 2016. It includes patient demographics, medical history, and—crucially—ground-truth resistance profiles for four common antibiotics.
  • Synthetic Dataset: A controlled environment where the relationship between patient features and drug success is highly non-linear (complex), but the optimal treatment rule remains a simple linear function. This was used to stress-test the direct vs. indirect approaches.

Key Insights and Ideas

  • The Pareto Frontier: There is no single "best" policy; there is a frontier of optimal trade-offs. By varying the reward weight $\omega$, the authors could generate a range of policies, allowing healthcare providers to choose their preferred balance between effectiveness and stewardship.
  • Decoupling Complexity: The synthetic experiments proved that direct policy learning is more robust. It can find a simple, effective decision rule even when the underlying biological outcomes are too messy for standard predictive models to capture accurately.
  • Smart Deferral: As the model was allowed to defer more cases to doctors, the performance gap on the remaining "decision cohort" actually widened. This suggests the model successfully identifies the specific types of cases where human intuition fails and algorithmic precision is most needed.

Key Results

The ML policies consistently landed "below and to the left" of human performance, meaning they achieved lower IAT rates and lower broad-spectrum usage simultaneously.

Figure 1: Performance frontier of learned policies compared to clinician performance and baselines. Policies with perfor- mance toward the bottom left of the figure are better. Our three approaches - thresholding, expected reward maximiza- tion, and direct learning - outperform clinicians and the con- strained baseline in [31]. Y-axis = 2nd-line usage rate, X-axis = inappropriate antibiotic therapy (IAT) rate.
  • Head-to-Head vs. Doctors: One learned policy reduced the IAT rate by 25% relative to clinicians while slightly reducing broad-spectrum usage.
  • Stewardship Win: Another policy virtually eliminated 2nd-line drug usage (reducing it from 33.6% to ~1%) while still reducing the IAT rate by 10% compared to doctors.
  • Synthetic Superiority: In complex environments, the direct approach approached Bayes-optimal performance with far fewer samples than the indirect approach.

Limitations and Future Work

  • Nonstationarity: Antibiotic resistance patterns change over time (e.g., a drug that worked in 2010 might be useless in 2024). The authors suggest future work should explicitly model these shifting trends.
  • Interpretability: While the linear direct policy is somewhat interpretable via its coefficients, more work is needed to make these "black box" suggestions truly transparent for frontline clinicians.
  • Data Assumptions: The "fully observed" setting is powerful but specific. Extending these multi-objective direct methods to "partially observed" settings (where we only know the outcome of the drug actually prescribed) remains a major frontier in causal inference.

Summary updated

Abstract

In several medical decision-making problems, such as antibiotic prescription, laboratory testing can provide precise indications for how a patient will respond to different treatment options. This enables us to "fully observe" all potential treatment outcomes, but while present in historical data, these results are infeasible to produce in real-time at the point of the initial treatment decision. Moreover, treatment policies in these settings often need to trade off between multiple competing objectives, such as effectiveness of treatment and harmful side effects. We present, compare, and evaluate three approaches for learning individualized treatment policies in this setting: First, we consider two indirect approaches, which use predictive models of treatment response to construct policies optimal for different trade-offs between objectives. Second, we consider a direct approach that constructs such a set of policies without intermediate models of outcomes. Using a medical dataset of Urinary Tract Infection (UTI) patients, we show that all approaches learn policies that achieve strictly better performance on all outcomes than clinicians, while also trading off between different objectives. We demonstrate additional benefits of the direct approach, including flexibly incorporating other goals such as deferral to physicians on simple cases.
BibTeX
@inproceedings{boominathan2020treatment,
  author = {Sooraj Boominathan and Michael Oberst and Helen Zhou and Sanjat Kanjilal and David A. Sontag},
  title = {{Treatment Policy Learning in Multiobjective Settings with Fully Observed Outcomes}},
  booktitle = {ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  year = {2020},
  doi = {10.1145/3394486.3403245},
}
Figure