Evaluating Algorithmic Bias in 30-Day Hospital Readmission Models: Retrospective Analysis

H. Echo Wang, Jonathan P. Weiner, Suchi Saria, Hadi KharraziJournal of Medical Internet Research 2024 · DOI

TL;DR

This paper investigates algorithmic bias in common 30-day hospital readmission models, finding that while retrained models and the modified HOSPITAL score show better performance and lower bias, interpreting fairness metrics requires careful consideration of real-world healthcare complexities. The study highlights that these metrics are a crucial first step for detecting disparities but are insufficient on their own to address the root causes of health inequity.

What Problem Does This Paper Address?

The increasing integration of predictive algorithms in healthcare, particularly for tasks like predicting hospital readmissions, raises concerns about algorithmic bias. This bias can exacerbate existing health disparities, leading to unequal allocation of resources and potentially worsening outcomes for marginalized groups. While fairness metrics exist to quantify this bias, their application and interpretation in real-world healthcare scenarios, especially for complex outcomes like hospital readmissions, remain limited. This study aims to bridge this gap by evaluating common readmission models using fairness metrics and assessing the practical utility and interpretability of these metrics in identifying and understanding disparities.

Methods

The researchers conducted a retrospective analysis using a large dataset of adult inpatient discharges from Maryland and Florida. They evaluated three common 30-day hospital readmission models: the LACE Index, the modified HOSPITAL score, and the modified Centers for Medicare & Medicaid Services (CMS) readmission measure. These models were applied both as-is (using existing coefficients) and retrained (recalibrated on a portion of the data) to mimic real-world implementation.

Key to their approach was the evaluation of predictive performance using metrics like Area Under the Curve (AUC) and Brier score, and importantly, the assessment of algorithmic bias. Bias was measured using several fairness metrics, including:

  • False Negative Rate (FNR) difference: The disparity in the rate of patients incorrectly predicted as low-risk for readmission.
  • False Positive Rate (FPR) difference: The disparity in the rate of patients incorrectly predicted as high-risk for readmission.
  • 0-1 loss difference: The disparity in overall misclassification rates.
  • Generalized Entropy Index (GEI): A measure of income inequality.

These measures were calculated for the overall population and then stratified by race (Black vs. White) and income (low-income vs. other-income) to identify differential performance. The study also performed stratification by hospital characteristics and patient population composition to understand how bias manifests in different contexts.

Datasets

The study utilized data from the State Inpatient Databases (SIDs) as part of the Healthcare Cost and Utilization Project (HCUP). This included approximately 1.9 million adult inpatient discharges from Maryland and 8.7 million from Florida, spanning the years 2016 to 2019. The SIDs contain comprehensive longitudinal hospital care data, including information on insurance payers and patient demographics. This dataset is a large-scale, real-world source of healthcare utilization data.

Key Insights and Ideas

  • Retraining Matters: Retraining or recalibrating models with local data significantly improved both predictive performance and reduced algorithmic bias, especially for the CMS measure. This underscores the importance of adapting models to specific populations and healthcare environments.
  • HOSPITAL Score Shines: The modified HOSPITAL score consistently demonstrated good calibration and relatively low bias across various measures, making it a strong contender among the evaluated models.
  • Fairness Metrics Aren't Always Intuitive: The study emphasizes that interpreting fairness metrics at face value can be misleading. For instance, a higher FNR for certain groups might indicate missed intervention opportunities, but it could also reflect different healthcare-seeking behaviors or access gaps.
  • Context is King: Algorithmic bias is not a monolithic issue. It varies significantly by hospital, population composition, and the specific fairness metric used. This highlights the need for nuanced interpretation rather than a one-size-fits-all approach.
  • Bias as a Starting Point, Not an End Goal: Fairness metrics are valuable for detecting disparate model performance and initiating discussions, but they are insufficient to explain the underlying causes of health disparities or to guide policy changes.

Key Results

  • The retrained CMS model achieved the best predictive performance (AUC of 0.74 in Maryland and 0.68-0.70 in Florida).
  • The modified HOSPITAL score showed the best calibration (Brier score of 0.16-0.19 in Maryland and 0.19-0.21 in Florida).
  • In Maryland, the retrained CMS and modified HOSPITAL score exhibited the lowest racial and income bias.
  • In Florida, both the retrained CMS and modified HOSPITAL score had the lowest income bias, with the modified HOSPITAL score also showing the lowest racial bias.
  • Across both states, White and higher-income populations tended to have a higher FNR (more false negatives), suggesting potential missed interventions.
  • Conversely, Black and low-income populations had a higher FPR (more false positives) and higher 0-1 loss, indicating more incorrect high-risk predictions.
  • Stratification by hospital revealed heterogeneous algorithmic bias, with some models showing more consistent performance across hospitals than others.

Limitations and Future Work

The authors acknowledge several limitations:

  • Model Accuracy: The evaluated models had moderate accuracy, which can impact the measurement of misclassifications.
  • Model Optimization: The study focused on model application and bias identification, not on optimizing models to minimize bias during training.
  • Algorithm-Agnostic Measures: Only algorithm-agnostic bias measures were included, meaning a comprehensive assessment of all potential biases was not possible.
  • Geographic Scope: The findings are based on Maryland and Florida, which may not represent the national landscape due to differences in healthcare systems and demographics.
  • Data Granularity: The administrative data lacked detailed clinical information (e.g., medications, lab results) that could provide deeper insights into the drivers of readmissions and bias.

Future work should focus on:

  • Improving the methodology for measuring algorithmic bias.
  • Testing more fairness definitions and measures, particularly those that consider calibration and predictive parity.
  • Exploring how modeling factors (variable inclusion, weights, scoring schemes) influence algorithmic bias.
  • Developing a more comprehensive approach that integrates predictive accuracy, algorithmic bias, and an understanding of the underlying healthcare system to inform meaningful actions that reduce health care disparity.

Summary updated

Abstract

Caution must be taken when interpreting fairness measures' face value. A higher FNR or FPR could potentially reflect missed opportunities or wasted resources, but these measures could also reflect health care use patterns and gaps in care. Simply relying on the statistical notions of bias could obscure or underplay the causes of health disparity. The imperfect health data, analytic frameworks, and the underlying health systems must be carefully considered. Fairness measures can serve as a useful routine assessment to detect disparate model performances but are insufficient to inform mechanisms or policy changes. However, such an assessment is an important first step toward data-driven improvement to address existing health disparities.
BibTeX
@article{wang2024evaluating,
  author = {H. Echo Wang and Jonathan P. Weiner and Suchi Saria and Hadi Kharrazi},
  title = {{Evaluating Algorithmic Bias in 30-Day Hospital Readmission Models: Retrospective Analysis}},
  journal = {Journal of Medical Internet Research},
  year = {2024},
  doi = {10.2196/47125},
}
Figure