Treatment Policy Learning in Multiobjective Settings with Fully Observed Outcomes

Soorajnath Boominathan, Michael Oberst, Helen Zhou, Sanjat Kanjilal, David A. Sontag

Abstract

Medical decision-making often involves balancing competing objectives, such as maximizing treatment efficacy while minimizing the use of broad-spectrum antibiotics to combat resistance. This paper investigates treatment policy learning in settings where historical data provides fully observed outcomes for all possible actions—a common scenario in clinical diagnostics like antibiotic susceptibility testing. We compare indirect predictive modeling against direct policy optimization. Our results on a real-world Urinary Tract Infection (UTI) dataset demonstrate that machine learning models can significantly outperform clinicians, reducing inappropriate therapy by over 25% at equivalent stewardship levels. Furthermore, direct policy optimization proves more sample-efficient in complex environments, offering a robust framework for multi-objective clinical decision support.

Introduction

In many medical scenarios, such as treating a Urinary Tract Infection (UTI), doctors face a "goldilocks" dilemma. They must choose between a narrow-spectrum antibiotic (targeted, lower risk of resistance) or a broad-spectrum antibiotic (higher efficacy, but contributes to "superbugs"). While precise lab results take days to process, historical datasets often contain these delayed results for every drug that could have been prescribed. This paper explores how to translate this "perfect hindsight" into real-time decision policies that navigate the trade-off between immediate patient health and long-term antibiotic stewardship.

Methodology

The authors evaluate three distinct approaches to learning a "Pareto frontier" of policies:

  1. Indirect Thresholding: Models predict the success probability for each drug; the policy selects the narrowest drug meeting a specific threshold.
  2. Indirect Expected Reward Maximization: Effectiveness and stewardship "costs" are combined into a single weighted reward score.
  3. Direct Policy Optimization: This method directly optimizes the decision rule using a multinomial deviance loss. By focusing on the decision boundary rather than modeling every nuance of patient health, this approach can be more robust and data-efficient.
Figure 5: Sample efficiency comparison on synthetic data. The Direct approach reaches near-optimal performance with significantly fewer samples than indirect predictive methods.

Additionally, the authors introduce a Learning to Defer framework, allowing the model to identify cases where clinicians are likely to err and only intervening then to mitigate alert fatigue.

Experimental Results

The models were evaluated on a real-world UTI dataset of 15,806 specimens and a synthetic dataset designed to test decision boundary complexity.

Beating the Experts

All machine learning approaches achieved strictly better performance than clinicians. When matched to the same broad-spectrum usage rate as doctors (33.6%), the ML models reduced the rate of inappropriate therapy (IAT) by over 25% (dropping from 11.9% to ~8.9%).

Figure 1: The Pareto frontier of IAT rate vs. 2nd-line usage. All ML methods (Direct, Expected Reward, Thresholding) achieve a superior trade-off compared to the clinician baseline (Doctor).

Stewardship and Efficiency

The models demonstrated the ability to virtually eliminate broad-spectrum drug usage (reducing it to <1%) while still providing better patient outcomes than current clinical practice. Furthermore, synthetic tests confirmed that the Direct approach is significantly more sample-efficient when the underlying biology is complex but the optimal decision boundary is relatively simple.

Discussion and Limitations

A primary limitation is the "fully observed" assumption; many medical fields do not provide counterfactual outcomes for all possible paths. Additionally, antibiotic resistance is nonstationary; future work must address how to update these policies as bacterial resistance patterns evolve over time. Despite these challenges, the study provides a powerful framework for institutionalizing stewardship goals through data-driven policy learning.

Summary updated

Abstract

In several medical decision-making problems, such as antibiotic prescription, laboratory testing can provide precise indications for how a patient will respond to different treatment options. This enables us to "fully observe" all potential treatment outcomes, but while present in historical data, these results are infeasible to produce in real-time at the point of the initial treatment decision. Moreover, treatment policies in these settings often need to trade off between multiple competing objectives, such as effectiveness of treatment and harmful side effects. We present, compare, and evaluate three approaches for learning individualized treatment policies in this setting: First, we consider two indirect approaches, which use predictive models of treatment response to construct policies optimal for different trade-offs between objectives. Second, we consider a direct approach that constructs such a set of policies without intermediate models of outcomes. Using a medical dataset of Urinary Tract Infection (UTI) patients, we show that all approaches learn policies that achieve strictly better performance on all outcomes than clinicians, while also trading off between different objectives. We demonstrate additional benefits of the direct approach, including flexibly incorporating other goals such as deferral to physicians on simple cases.
BibTeX
@inproceedings{boominathan2020treatment,
  author = {Soorajnath Boominathan and Michael Oberst and Helen Zhou and Sanjat Kanjilal and David A. Sontag},
  title = {{Treatment Policy Learning in Multiobjective Settings with Fully Observed Outcomes}},
  booktitle = {ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  year = {2020},
  doi = {10.1145/3394486.3403245},
}
Figure