Abstract
Multiple reporting guidelines for artificial intelligence (AI) models in healthcare recommend that models be audited for reliability and fairness. However, there is a significant gap in operational guidance for performing these audits in practice. This study describes a reliability and fairness audit of two mortality prediction models—the Epic End-of-Life (EOL) Index and an internally developed Stanford Hospital Medicine (HM) Advance Care Planning (ACP) model—across three clinical settings: Primary Care, Inpatient Oncology, and Hospital Medicine. Using clinicians’ answers to the "surprise question" as a surrogate outcome, the audit revealed that while both models maintained high positive predictive value, the Stanford model demonstrated superior sensitivity and calibration. Crucially, intersectional analysis identified performance drops for Hispanic/Latino male patients. The audit required 115 person-hours, highlighting the operational costs of responsible AI deployment.
Introduction
As artificial intelligence (AI) becomes increasingly integrated into clinical workflows, the demand for rigorous auditing has grown. Guidelines like TRIPOD-AI and CLAIM emphasize the necessity of evaluating models for reliability and fairness before deployment. Yet, for many health systems, a significant gap remains between these high-level recommendations and the operational reality of executing an audit within a complex hospital environment.
This paper provides an "under the hood" look at the process of auditing AI models used for Advance Care Planning (ACP). In ACP, clinicians identify patients who may benefit from end-of-life care discussions. The stakes are high: an unreliable model might miss patients in need of support, while an unfair model could systematically exclude marginalized groups from timely care.
Methodology
Models Under Review
The researchers compared two distinct approaches to mortality prediction:
- Epic End-of-Life (EOL) Index: A proprietary logistic regression model utilizing 46 features.
- Stanford HM ACP Model: An internally developed gradient-boosted tree model utilizing 13,189 features.
The "Surprise Question" as a Surrogate Outcome
A primary challenge in auditing mortality models is the time lag; waiting 12 months for mortality data to mature is often impractical for iterative model evaluation. To address this, the authors used the "Surprise Question" (SQ): "Would you be surprised if [patient X] passed away in [Y years]?" Clinician "No" answers were treated as the ground truth label for high-risk patients, allowing for a real-time assessment of model-clinician alignment.
Audit Framework
The audit evaluated two primary dimensions:
- Reliability: Measured through sensitivity, specificity, Positive Predictive Value (PPV), and calibration (Observed/Expected ratios).
- Fairness: Evaluated through subgroup analysis (age, sex, race/ethnicity) and intersectional analysis, which examines combined identities (e.g., Hispanic/Latino Males) to uncover biases that aggregate statistics might hide.
Key Findings
Performance and Calibration
Both models demonstrated a Positive Predictive Value (PPV) at or above 0.76 across all settings, meaning that when a model flagged a patient, clinicians usually agreed. However, significant differences emerged in sensitivity and calibration:
- Sensitivity: In Hospital Medicine and Inpatient Oncology, the Stanford HM ACP model showed much higher sensitivity (0.69 and 0.89, respectively) compared to the Epic EOL model (0.20 and 0.27).
- Calibration: The Stanford model was better calibrated (O/E 1.5–1.7) than the Epic model (O/E 2.5–3.0). The Epic model flagged significantly fewer patients (11–21%) than the Stanford model (38–75%), suggesting it may be overly conservative in identifying high-risk individuals.
Intersectional Fairness
While no significant performance differences were found when looking at sex in isolation, the intersectional analysis revealed a critical gap. Both models exhibited lower sensitivity in Hispanic/Latino male patients whose race was listed as "Other." In the Inpatient Oncology setting, the Epic model failed to flag any of the 13 positive patients in this subgroup (sensitivity of 0.0). This finding underscores that aggregate data often masks localized "pockets" of unfairness.
The Operational Burden
One of the study's most striking findings was the effort required to conduct a thorough audit. The process took 115 person-hours over 8–10 months. The most time-consuming tasks included data processing and analysis (48 hours) and gathering clinician labels (24 hours). This represents a significant "hidden cost" of responsible AI that health systems must account for in their operational budgets.
Clinician Perspectives
A survey of 10 clinicians following the audit presentation revealed high engagement with the results. All 10 reported that seeing subgroup performance would affect their decision to use a model, and 9 out of 10 said the same for calibration data. This suggests that fairness and reliability audits are not just regulatory hurdles but are essential for establishing the clinical trust necessary for model adoption.
Recommendations and Conclusion
Based on the audit experience, the authors offer several recommendations for healthcare organizations:
- Prioritize Intersectional Analysis: Evaluating race or sex independently is insufficient to uncover specific algorithmic biases.
- Verify Data Validity: Poor demographic data quality in Electronic Health Records (EHRs) remains a primary barrier to accurate fairness auditing.
- Plan for the Cost: Reliability and fairness audits should be viewed as a standard part of the AI lifecycle, requiring dedicated time, data access, and stakeholder mediation.
By bridging the gap between theoretical guidelines and clinical practice, this study provides a roadmap for other institutions to implement routine, rigorous audits, ensuring that AI tools in healthcare are both effective and equitable.
Summary updated