Considerations in the Reliability and Fairness Audits of Predictive Models for Advance Care Planning

Jonathan Lu, Amelia Sattler, Samantha Wang, Ali Raza Khaki, Alison Callahan, Scott L. Fleming, Rebecca Fong, Benjamin Ehlert, Ron Li, Lisa Shieh, Kavitha Ramchandran, Michael F. Gensheimer, Sarah Chobot, Stephen Pfohl, Siyun Li, Kenny Shum, Nitin Parikh, Priya Desai, Briththa Seevaratnam, Melanie Hanson, Margaret Smith, Yizhe Xu, Arjun Gokhale, Steven Lin, Michael A. Pfeffer, Winifred Teuteberg, Nigam H. Shah

Abstract

Multiple reporting guidelines for artificial intelligence (AI) models in healthcare recommend that models be audited for reliability and fairness. However, there is a significant gap in operational guidance for performing these audits in practice. This study describes a reliability and fairness audit of two mortality prediction models—the Epic End-of-Life (EOL) Index and an internally developed Stanford Hospital Medicine (HM) Advance Care Planning (ACP) model—across three clinical settings: Primary Care, Inpatient Oncology, and Hospital Medicine. Using clinicians’ answers to the "surprise question" as a surrogate outcome, the audit revealed that while both models maintained high positive predictive value, the Stanford model demonstrated superior sensitivity and calibration. Crucially, intersectional analysis identified performance drops for Hispanic/Latino male patients. The audit required 115 person-hours, highlighting the operational costs of responsible AI deployment.

Introduction

As artificial intelligence (AI) becomes increasingly integrated into clinical workflows, the demand for rigorous auditing has grown. Guidelines like TRIPOD-AI and CLAIM emphasize the necessity of evaluating models for reliability and fairness before deployment. Yet, for many health systems, a significant gap remains between these high-level recommendations and the operational reality of executing an audit within a complex hospital environment.

This paper provides an "under the hood" look at the process of auditing AI models used for Advance Care Planning (ACP). In ACP, clinicians identify patients who may benefit from end-of-life care discussions. The stakes are high: an unreliable model might miss patients in need of support, while an unfair model could systematically exclude marginalized groups from timely care.

Methodology

Models Under Review

The researchers compared two distinct approaches to mortality prediction:

  1. Epic End-of-Life (EOL) Index: A proprietary logistic regression model utilizing 46 features.
  2. Stanford HM ACP Model: An internally developed gradient-boosted tree model utilizing 13,189 features.

The "Surprise Question" as a Surrogate Outcome

A primary challenge in auditing mortality models is the time lag; waiting 12 months for mortality data to mature is often impractical for iterative model evaluation. To address this, the authors used the "Surprise Question" (SQ): "Would you be surprised if [patient X] passed away in [Y years]?" Clinician "No" answers were treated as the ground truth label for high-risk patients, allowing for a real-time assessment of model-clinician alignment.

Audit Framework

The audit evaluated two primary dimensions:

  • Reliability: Measured through sensitivity, specificity, Positive Predictive Value (PPV), and calibration (Observed/Expected ratios).
  • Fairness: Evaluated through subgroup analysis (age, sex, race/ethnicity) and intersectional analysis, which examines combined identities (e.g., Hispanic/Latino Males) to uncover biases that aggregate statistics might hide.

Key Findings

Performance and Calibration

Both models demonstrated a Positive Predictive Value (PPV) at or above 0.76 across all settings, meaning that when a model flagged a patient, clinicians usually agreed. However, significant differences emerged in sensitivity and calibration:

  • Sensitivity: In Hospital Medicine and Inpatient Oncology, the Stanford HM ACP model showed much higher sensitivity (0.69 and 0.89, respectively) compared to the Epic EOL model (0.20 and 0.27).
  • Calibration: The Stanford model was better calibrated (O/E 1.5–1.7) than the Epic model (O/E 2.5–3.0). The Epic model flagged significantly fewer patients (11–21%) than the Stanford model (38–75%), suggesting it may be overly conservative in identifying high-risk individuals.

Intersectional Fairness

While no significant performance differences were found when looking at sex in isolation, the intersectional analysis revealed a critical gap. Both models exhibited lower sensitivity in Hispanic/Latino male patients whose race was listed as "Other." In the Inpatient Oncology setting, the Epic model failed to flag any of the 13 positive patients in this subgroup (sensitivity of 0.0). This finding underscores that aggregate data often masks localized "pockets" of unfairness.

The Operational Burden

One of the study's most striking findings was the effort required to conduct a thorough audit. The process took 115 person-hours over 8–10 months. The most time-consuming tasks included data processing and analysis (48 hours) and gathering clinician labels (24 hours). This represents a significant "hidden cost" of responsible AI that health systems must account for in their operational budgets.

Clinician Perspectives

A survey of 10 clinicians following the audit presentation revealed high engagement with the results. All 10 reported that seeing subgroup performance would affect their decision to use a model, and 9 out of 10 said the same for calibration data. This suggests that fairness and reliability audits are not just regulatory hurdles but are essential for establishing the clinical trust necessary for model adoption.

Recommendations and Conclusion

Based on the audit experience, the authors offer several recommendations for healthcare organizations:

  1. Prioritize Intersectional Analysis: Evaluating race or sex independently is insufficient to uncover specific algorithmic biases.
  2. Verify Data Validity: Poor demographic data quality in Electronic Health Records (EHRs) remains a primary barrier to accurate fairness auditing.
  3. Plan for the Cost: Reliability and fairness audits should be viewed as a standard part of the AI lifecycle, requiring dedicated time, data access, and stakeholder mediation.

By bridging the gap between theoretical guidelines and clinical practice, this study provides a roadmap for other institutions to implement routine, rigorous audits, ensuring that AI tools in healthcare are both effective and equitable.

Summary updated

Abstract

Abstract Multiple reporting guidelines for artificial intelligence (AI) models in healthcare recommend that models be audited for reliability and fairness. However, there is a gap of operational guidance for performing reliability and fairness audits in practice. Following guideline recommendations, we conducted a reliability audit of two models based on model performance and calibration as well as a fairness audit based on summary statistics, subgroup performance and subgroup calibration. We assessed the Epic End-of-Life (EOL) Index model and an internally developed Stanford Hospital Medicine (HM) Advance Care Planning (ACP) model in 3 practice settings: Primary Care, Inpatient Oncology and Hospital Medicine, using clinicians’ answers to the surprise question (“Would you be surprised if [patient X] passed away in [Y years]?”) as a surrogate outcome. For performance, the models had positive predictive value (PPV) at or above 0.76 in all settings. In Hospital Medicine and Inpatient Oncology, the Stanford HM ACP model had higher sensitivity (0.69, 0.89 respectively) than the EOL model (0.20, 0.27), and better calibration (O/E 1.5, 1.7) than the EOL model (O/E 2.5, 3.0). The Epic EOL model flagged fewer patients (11%, 21% respectively) than the Stanford HM ACP model (38%, 75%). There were no differences in performance and calibration by sex. Both models had lower sensitivity in Hispanic/Latino male patients with Race listed as “Other.” 10 clinicians were surveyed after a presentation summarizing the audit. 10/10 reported that summary statistics, overall performance, and subgroup performance would affect their decision to use the model to guide care; 9/10 said the same for overall and subgroup calibration. The most commonly identified barriers for routinely conducting such reliability and fairness audits were poor demographic data quality and lack of data access. This audit required 115 person-hours across 8-10 months. Our recommendations for performing reliability and fairness audits include verifying data validity, analyzing model performance on intersectional subgroups, and collecting clinician-patient linkages as necessary for label generation by clinicians. Those responsible for AI models should require such audits before model deployment and mediate between model auditors and impacted stakeholders. Contribution to the Field Statement Artificial intelligence (AI) models developed from electronic health record (EHR) data can be biased and unreliable. Despite multiple guidelines to improve reporting of model fairness and reliability, adherence is difficult given the gap between what guidelines seek and operational feasibility of such reporting. We try to bridge this gap by describing a reliability and fairness audit of AI models that were considered for use to support team-based advance care planning (ACP) in three practice settings: Primary Care, Inpatient Oncology, and Hospital Medicine. We lay out the data gathering processes as well as the design of the reliability and fairness audit, and present results of the audit and decision maker survey. We discuss key lessons learned, how long the audit took to perform, requirements regarding stakeholder relationships and data access, and limitations of the data. Our work may support others in implementing routine reliability and fairness audits of models prior to deployment into a practice setting.
BibTeX
@article{lu2022considerations,
  author = {Jonathan Lu and Amelia Sattler and Samantha Wang and Ali Raza Khaki and Alison Callahan and Scott L. Fleming and Rebecca Fong and Benjamin Ehlert and Ron Li and Lisa Shieh and Kavitha Ramchandran and Michael F. Gensheimer and Sarah Chobot and Stephen Pfohl and Siyun Li and Kenny Shum and Nitin Parikh and Priya Desai and Briththa Seevaratnam and Melanie Hanson and Margaret Smith and Yizhe Xu and Arjun Gokhale and Steven Lin and Michael A. Pfeffer and Winifred Teuteberg and Nigam H. Shah},
  title = {{Considerations in the Reliability and Fairness Audits of Predictive Models for Advance Care Planning}},
  journal = {Frontiers Digit. Health},
  year = {2022},
  doi = {10.3389/fdgth.2022.943768},
}
Figure