Machine learning is more accurate and biased than risk scoring tools in the prediction of postoperative atrial fibrillation after cardiac surgery

Joyce C. Ho, Shalmali Joshi, Eduardo Valverde, Kathryn J. Wood, Kendra J. Grubb, Miguel Leal, Vicki HertzbergmedRxiv 2024 · DOI

Predicting complications after heart surgery is a high-stakes game. One of the most common hurdles is postoperative atrial fibrillation (POAF)—a heart rhythm disorder that affects up to half of all cardiac surgery patients and leads to longer hospital stays, higher costs, and increased mortality.

A recent study published in medRxiv explores a modern dilemma: while machine learning (ML) is significantly better at predicting who will develop POAF than traditional clinical risk scores, it often does so at the expense of fairness, particularly across racial lines.

TL;DR

Machine learning models significantly outperform traditional clinical risk scores in predicting atrial fibrillation after cardiac surgery. However, these gains in accuracy come with a "fairness tax," as ML models exhibit substantially more bias against Black patients compared to simpler, rule-based scoring tools.

What Problem Does This Paper Address?

For decades, clinicians have used "risk scores"—simple formulas like CHADS₂ or HATCH—to identify high-risk patients. While easy to use, these tools often have "moderate" predictive power, frequently failing to generalize across diverse populations.

Machine learning offers a more powerful alternative, capable of parsing thousands of data points from Electronic Health Records (EHRs). However, if an ML model is trained on biased data or fails to account for subpopulation differences, it can exacerbate existing healthcare disparities. The authors set out to determine if the superior performance of ML holds up in a diverse patient cohort and, crucially, whether that performance is equitable across race and sex.

Methods

The researchers conducted a retrospective study comparing 8 traditional risk scores against 6 popular ML algorithms:

  • ML Models: Logistic Regression (LR), Decision Trees (DT), Support Vector Machines (SVM), Random Forests (RF), Gradient Boosting Machines (GBM), and Multi-layer Perceptrons (MLP).
  • Evaluation: They used 10-fold stratified Monte Carlo cross-validation to ensure robust results.
  • Fairness Metrics: Beyond standard accuracy (AUROC/AUPRC), they employed two key fairness metrics:
  • Demographic Parity Ratio (DPR): Does the model predict the same proportion of POAF across groups?
  • Equalized Odds Ratio (EQR): Are the true positive and false positive rates the same across groups?

Datasets

The study utilized a de-identified EHR dataset from the Emory Healthcare clinical data warehouse.

  • Cohort: 4,961 adult patients who underwent cardiac surgery between 2013 and 2017.
  • Diversity: Unlike many previous studies where Black patients represented less than 4% of the data, this cohort was significantly more diverse: 21.9% Black, 69.1% White, and 9.1% Other.
  • Features: 71 variables were extracted, including demographics, vital signs, and clinical codes (grouped via CCS and ATC systems).

Key Insights and Ideas

  • The Performance Gap is Real: Every single ML model tested outperformed the best-performing clinical risk score (CHARGE-AF). This suggests that the complex, non-linear relationships captured by ML are vital for predicting POAF.
  • The Fairness Trade-off: There is a clear inverse relationship between accuracy and fairness. The "simplest" risk scores (which often ignore race or sex) were perfectly fair (DPR/EQR of 1.0) but lacked accuracy. The most accurate ML models were the least fair.
  • Race is the Primary Friction Point: While sex-based bias was present, racial bias was much more pronounced. ML models were consistently less accurate and less "fair" when predicting outcomes for Black patients compared to White patients.
  • Proxies for Bias: Even when race isn't the primary feature, ML models can pick up on "proxy" variables—like specific comorbidities or medication patterns—that correlate with race, leading to biased "algorithmic profiling" without the model ever "knowing" the patient's race.

Key Results

  • Top Performer: Random Forest (RF) achieved the highest discrimination with an AUROC of 0.671 and AUPRC of 0.558.
  • Risk Score Baseline: The best risk score, CHARGE-AF, trailed behind with an AUROC of 0.585.
  • Fairness Scores: Most traditional risk scores maintained a perfect fairness score of 1.000. In contrast, the top-performing RF model dropped to a DPR of 0.824 and EQR of 0.816 for race, indicating significant disparity in how it assesses risk for different racial groups.

Limitations and Future Work

The authors acknowledge several limitations:

  1. Data Scope: The study relied on structured EHR data. It excluded unstructured data like echocardiogram notes (e.g., left atrial diameter), which are known predictors of AF but are difficult to extract at scale.
  2. Single-Center Study: The data comes from one healthcare system (Emory), which may limit how well these specific models work in other geographic regions.
  3. Lack of Mitigation: While the study identifies the bias, it does not implement "de-biasing" techniques during the training phase.

Future Work: The researchers highlight an urgent need for "Fair ML" research in clinical settings—developing models that don't just chase the highest AUROC, but optimize for a balance of high performance and equitable treatment across all patient demographics.

Summary updated

Abstract

Abstract Incidence of postoperative atrial fibrillation (POAF) after cardiac surgery remains high and is associated with adverse patient outcomes. Risk scoring tools have been developed to predict POAF, yet discrimination performance remains moderate. Machine learning (ML) models can achieve better performance but may exhibit performance heterogeneity across race and sex subpopulations. We evaluate 8 risk scoring tools and 6 ML models on a heterogeneous cohort derived from electronic health records. Our results suggest that ML models achieve higher discrimination yet are less fair, especially with respect to race. Our findings highlight the need for building accurate and fair ML models to facilitate consistent and equitable assessment of POAF risk.
BibTeX
@misc{ho2024machine,
  author = {Joyce C. Ho and Shalmali Joshi and Eduardo Valverde and Kathryn J. Wood and Kendra J. Grubb and Miguel Leal and Vicki Hertzberg},
  title = {{Machine learning is more accurate and biased than risk scoring tools in the prediction of postoperative atrial fibrillation after cardiac surgery}},
  year = {2024},
  doi = {10.1101/2024.07.05.24310013},
  howpublished = {medRxiv},
}
Figure