Abstract
Objective: To analyze the impact of factors in healthcare delivery on the net benefit of triggering an Advanced Care Planning (ACP) workflow based on predictions of 12-month mortality.
Materials and Methods: We built a predictive model of 12-month mortality using electronic health record data and evaluated the impact of healthcare delivery factors on the net benefit of triggering an ACP workflow based on the models’ predictions. Factors included non-clinical reasons that make ACP inappropriate, limited capacity for ACP, inability to follow up due to patient discharge, and availability of an outpatient workflow to follow up on missed cases.
Results: Work capacity constraints and discharge timing can significantly reduce the net benefit of triggering the ACP workflow. However, the reduction can be mitigated by creating an outpatient ACP workflow.
Conclusion: An analysis of the sensitivity of the net benefit realized by a predictive model to various healthcare delivery factors is necessary for making predictive models useful in practice.
Introduction: The Implementation Gap
In the current landscape of healthcare AI, there is a stark contrast between the abundance of high-performing predictive models and the scarcity of those that actually improve patient outcomes. This "implementation gap" often stems from a focus on technical metrics like AUROC while ignoring "last-mile" delivery constraints. This paper presents a framework to bridge this gap by incorporating operational realities—such as staff capacity and patient discharge timing—into the evaluation of clinical AI.
The Case for Advance Care Planning (ACP)
The authors use Advance Care Planning (ACP) as a primary case study. ACP involves documenting a patient's end-of-life wishes, a process known to improve care quality and reduce unnecessary costs. Even with a perfect model identifying high-risk patients, the actual delivery of ACP is limited by hospital resources. The paper provides a framework to quantify how these real-world "failure modes" erode the theoretical benefit of predictive models.
Methodology: A Four-Stage Framework
The researchers propose a four-stage deployment framework:
- Problem Specification: Defining the clinical target (12-month mortality) and the intervention (ACP).
- Technical Validation: Building the predictive model (a LightGBM model with 63,043 features achieving an AUROC of 0.86).
- Utility Analysis: Using simulations to quantify how workflow constraints impact "Net Utility."
- Monitoring: Ongoing evaluation of the system in practice.
Quantifying Utility
To move beyond abstract accuracy, the authors assigned utility values to the four possible outcomes of a prediction (True Positive, False Negative, False Positive, and True Negative). These values were derived from multi-site RCT data on the cost savings of palliative care, adjusted for inflation to reflect modern healthcare expenditures.
Simulating the "Last Mile"
The core of this research is a simulation of 5,000 days of hospital operations. Unlike standard evaluations, this simulation accounts for the dynamic nature of a hospital, where patients arrive at different times and staff have limited bandwidth. The researchers varied parameters such as daily capacity and the rate at which patients might decline an intervention for non-clinical reasons.
Key Insights: Beyond Model Accuracy
The study revealed several critical insights that challenge the traditional "better model = better care" paradigm:
- Capacity as a Bottleneck: A model with an AUROC of 0.86 can lose over 50% of its potential utility if the hospital can only perform one ACP per day. Clinical benefit is non-linear; increasing staff capacity yields diminishing returns because high-risk admissions are often clustered on specific days.
- The Impact of Discharge: If an ACP takes two days to complete on average, approximately 25% of the potential benefit is lost because patients are discharged before the conversation can be finalized.
- The Power of Outpatient "Rescue": Perhaps the most significant finding was that an outpatient pathway to catch patients missed during their hospital stay is more effective than simply hiring more inpatient staff. A rescue pathway with only a 50% success rate outperformed significant increases in inpatient capacity across all starting levels.
Conclusion
The utility of a predictive model is highly dependent on the capacity to execute the workflow it triggers. This framework provides a method for quantifying the impact of healthcare delivery factors before deployment. By shifting focus from model optimization to workflow optimization, healthcare systems can ensure that AI investments translate into tangible benefits for patient care and operational efficiency.
Reference: Jung, K., et al. (2020). A framework for making predictive models useful in practice. medRxiv preprint.
Summary updated