Considerations in the reliability and fairness audits of predictive models for advance care planning

Jonathan Lu, Amelia Sattler, Samantha Wang, Ali Raza Khaki, Alison Callahan, Scott L. Fleming, Rebecca Fong, Benjamin Ehlert, Ron C. Li, Lisa Shieh, Kavitha Ramchandran, Michael Gensheimer, Sarah Chobot, Stephen Pfohl, Siyun Li, Kenny Shum, Nitin Parikh, Priya Desai, Briththa Seevaratnam, Melanie Hanson, Margaret Smith, Yizhe Xu, Arjun Gokhale, Steven Lin, Michael A. Pfeffer, Winifred Teuteberg, Nigam H. Shah

Introduction

Predictive models in healthcare are increasingly used to guide critical decisions, such as identifying patients for Advance Care Planning (ACP). While guidelines like TRIPOD-AI and PROBAST-AI emphasize the need for reliability and fairness audits, a significant "implementation gap" exists between these theoretical principles and actual clinical practice. This study provides a practical roadmap for auditing healthcare AI by conducting a head-to-head assessment of two mortality prediction models, revealing the substantial operational effort required and the critical importance of intersectional subgroup analysis.

Methodology

The researchers evaluated two models designed to identify patients at high risk of 12-month mortality: the commercial Epic End-of-Life (EOL) Index (logistic regression) and an internally developed Stanford Hospital Medicine (HM) ACP model (gradient-boosted tree).

To assess these models, the team utilized the "Surprise Question" (SQ) as a ground truth surrogate: "Would you be surprised if [patient X] passed away in [Y years]?" Performance was measured through Positive Predictive Value (PPV), sensitivity, specificity, and calibration (Observed/Expected ratios). Crucially, the audit employed an intersectional fairness framework, examining performance across combined demographic categories (e.g., Hispanic Males) rather than isolated variables. Finally, the researchers surveyed 10 clinicians to understand how audit metrics influenced their trust in the AI tools.

Data and Settings

The study leveraged Electronic Health Record (EHR) data from Stanford Health Care, mapped to the OMOP Common Data Model. The evaluation set comprised 1,085 clinician-generated labels across Primary Care, Inpatient Oncology, and Hospital Medicine. Demographic data included age, sex, and race/ethnicity, with the latter treated as a social construct to acknowledge the limitations of EHR-recorded data.

Key Findings: Performance and Fairness

The audit revealed significant disparities between the two models. While both maintained high precision (PPV ≥ 0.76), the Stanford model demonstrated superior sensitivity and calibration. In Hospital Medicine, the Stanford model achieved a sensitivity of 0.69, catching nearly 70% of high-risk patients, whereas the Epic model caught only 20% (sensitivity 0.20).

Calibration plot for the Epic EOL model showing significant underprediction of risk across sexes.

Calibration was a major differentiator. The Epic model consistently underpredicted risk, with O/E ratios between 2.5 and 3.0, indicating that actual "surprising" deaths occurred far more frequently than predicted. In contrast, the Stanford model's O/E ratios (1.5 to 1.7) were much closer to the ideal 1.0. Clinician surveys confirmed that subgroup calibration is a vital metric for trust, with 90% stating it would influence their decision to use the tool.

The intersectional analysis uncovered critical "blind spots." For instance, the Epic model showed 0% sensitivity for Hispanic/Latino males in Inpatient Oncology—a failure that would have been hidden by aggregate statistics. Both models generally exhibited lower sensitivity for Hispanic/Latino male patients compared to the overall population.

The Operational "Tax"

A key takeaway for healthcare organizations is the massive operational effort required for a robust audit. The study documented a "tax" of 115 person-hours over 8–10 months.

The audit process required a significant operational "tax" of over 100 person-hours, involving data cleaning and clinician coordination.

The majority of this time was consumed by "data cascades"—complex tasks such as cleaning messy demographic data, linking clinicians to patients for labeling, and navigating institutional data access hurdles. This suggests that auditing is as much an organizational and informatics challenge as it is a computational one.

Conclusion and Future Directions

The study highlights that local tuning or custom model architecture may be necessary for high-stakes clinical tasks, as the internal Stanford model outperformed the commercial alternative. However, the use of surrogate outcomes like the "Surprise Question" and the inherent inaccuracies in EHR demographic data remain limitations.

Future efforts should focus on empowering "model auditors" to bridge the gap between developers and clinical stakeholders. Informatics teams must also prioritize building better clinician-patient linkage data to streamline the auditing process, making it a routine part of the AI lifecycle in healthcare.

Summary updated

Abstract

p>Multiple reporting guidelines for artificial intelligence (AI) models in healthcare recommend that models be audited for reliability and fairness. However, there is a gap of operational guidance for performing reliability and fairness audits in practice. Following guideline recommendations, we conducted a reliability audit of two models based on model performance and calibration as well as a fairness audit based on summary statistics, subgroup performance and subgroup calibration. We assessed the Epic End-of-Life (EOL) Index model and an internally developed Stanford Hospital Medicine (HM) Advance Care Planning (ACP) model in 3 practice settings: Primary Care, Inpatient Oncology and Hospital Medicine, using clinicians' answers to the surprise question (“Would you be surprised if [patient X] passed away in [Y years]?”) as a surrogate outcome. For performance, the models had positive predictive value (PPV) at or above 0.76 in all settings. In Hospital Medicine and Inpatient Oncology, the Stanford HM ACP model had higher sensitivity (0.69, 0.89 respectively) than the EOL model (0.20, 0.27), and better calibration (O/E 1.5, 1.7) than the EOL model (O/E 2.5, 3.0). The Epic EOL model flagged fewer patients (11%, 21% respectively) than the Stanford HM ACP model (38%, 75%). There were no differences in performance and calibration by sex. Both models had lower sensitivity in Hispanic/Latino male patients with Race listed as “Other.” 10 clinicians were surveyed after a presentation summarizing the audit. 10/10 reported that summary statistics, overall performance, and subgroup performance would affect their decision to use the model to guide care; 9/10 said the same for overall and subgroup calibration. The most commonly identified barriers for routinely conducting such reliability and fairness audits were poor demographic data quality and lack of data access. This audit required 115 person-hours across 8–10 months. Our recommendations for performing reliability and fairness audits include verifying data validity, analyzing model performance on intersectional subgroups, and collecting clinician-patient linkages as necessary for label generation by clinicians. Those responsible for AI models should require such audits before model deployment and mediate between model auditors and impacted stakeholders.</p>
BibTeX
@article{lu2022considerations,
  author = {Jonathan Lu and Amelia Sattler and Samantha Wang and Ali Raza Khaki and Alison Callahan and Scott L. Fleming and Rebecca Fong and Benjamin Ehlert and Ron C. Li and Lisa Shieh and Kavitha Ramchandran and Michael Gensheimer and Sarah Chobot and Stephen Pfohl and Siyun Li and Kenny Shum and Nitin Parikh and Priya Desai and Briththa Seevaratnam and Melanie Hanson and Margaret Smith and Yizhe Xu and Arjun Gokhale and Steven Lin and Michael A. Pfeffer and Winifred Teuteberg and Nigam H. Shah},
  title = {{Considerations in the reliability and fairness audits of predictive models for advance care planning}},
  journal = {Frontiers Digit. Health},
  year = {2022},
  doi = {10.3389/fdgth.2022.943768},
}
Figure