Comparison between machine learning methods for mortality prediction for sepsis patients with different social determinants

Hanyin Wang, Yikuan Li, Andrew Naidech, Yuan LuoBMC Medical Informatics Decis. Mak 2022 · DOI

TL;DR

This study investigates how social determinants (race, language, etc.) impact the performance of machine learning models for predicting sepsis patient mortality. The findings reveal that a "one-size-fits-all" approach is insufficient, as model performance significantly degrades for certain subpopulations, highlighting the need for more equitable and adaptable AI in healthcare.

What Problem Does This Paper Address?

Sepsis is a critical and life-threatening condition, but its diagnosis and management are complex. While machine learning (ML) holds promise for improving sepsis risk prediction, existing models are often trained on broad datasets without fully accounting for the diverse social determinants of health (SDOH) that can influence patient outcomes and data representation. These SDOH, such as race, ethnicity, language, insurance status, and marital status, can lead to disparities in healthcare access, treatment, and ultimately, model performance. This paper aims to bridge this gap by systematically evaluating how ML models trained on a general population perform when applied to specific subpopulations defined by these social determinants, uncovering potential biases and inequities.

Methods

The researchers employed a rigorous ML pipeline to address their research question. They utilized the Medical Information Mart for Intensive Care (MIMIC)-III database to identify a cohort of critical care patients. Sepsis patients were identified using the Sepsis-3 criteria, considered the most conservative and recent standard.

The core of their methodology involved:

  • Data Splitting: The identified sepsis patient cohort was split into training (70%) and testing (30%) sets.
  • Model Training: Sixteen different ML classifiers were trained on the training set to predict in-hospital mortality. These models included a variety of algorithms such as Ridge classifier, Perceptron, k-Nearest Neighbors (kNN), Random Forest, Support Vector Machines (SVMs) with different kernels and regularizations, and logistic regression.
  • Performance Evaluation: Model performance was primarily measured using the Area Under the Receiver Operating Characteristic Curve (AUC).
  • Subpopulation Analysis: Crucially, the trained models were then tested not only on the entire testing set but also on specific subpopulations defined by race, sex, marital status, insurance type, and language.
  • Disparity Detection: To quantify the performance differences, permutation tests (1000 iterations) were used. These tests compared the AUCs of the models on the entire testing set against their AUCs on each subpopulation. Pairwise permutation tests were also conducted to compare performance between specific social determinant groups.

Datasets

The study exclusively used the Medical Information Mart for Intensive Care (MIMIC)-III v1.4 database. This is a well-established, large-scale, publicly available database containing de-identified health-related data associated with patients who stayed in critical care units. The researchers filtered this database to include 11,791 patients with their initial admission records, and subsequently identified 5,783 sepsis patients based on the Sepsis-3 criteria for their analysis. No new benchmarks were introduced; the study leveraged existing data and established sepsis identification criteria.

Key Insights and Ideas

  • "One-Size-Fits-All" ML Fails for Sepsis Prediction: The most significant insight is that a single ML model trained on a general sepsis patient population does not perform equally well across all social determinant groups. This directly challenges the assumption that a universally optimized model is sufficient.
  • Race and Language are Key Drivers of Disparity: The study found statistically significant performance drops in mortality prediction for Asian and Hispanic patients, as well as for Spanish-speaking patients. This highlights that these specific social determinants have a pronounced impact on model accuracy.
  • Subtle but Significant Pairwise Differences: Even when overall performance might seem acceptable, pairwise comparisons revealed significant disparities. For instance, Asian patients showed performance discrepancies compared to White patients and patients of other races, and Spanish-speaking patients performed worse than English-speaking patients.
  • Sepsis Criteria Matter: The study noted that different sepsis identification criteria (e.g., Sepsis-3 vs. Angus methodology) yield varying proportions of patients within social determinant groups, suggesting that the choice of diagnostic criteria itself can be influenced by these factors.

Key Results

  • A total of 5,783 sepsis patients were identified using the Sepsis-3 criteria.
  • When tested on subpopulations, significant performance decreases were observed for Asian and Hispanic patients, and Spanish-speaking patients.
  • Pairwise comparisons revealed statistically significant performance discrepancies between:
  • Asian and White patients.
  • Asian and patients of other races.
  • English-speaking and Spanish-speaking patients.
  • While the paper doesn't provide a single "best" AUC, Table 2 shows that various classifiers achieved AUCs in the range of 0.64 to 0.78 on the entire testing set, indicating a baseline level of predictive capability. However, the subsequent subpopulation analyses demonstrated that this overall performance is not uniformly distributed.

Limitations and Future Work

  • Exclusion of Missing Data: The study acknowledges that patients with missing data were excluded, which could introduce bias. They propose using advanced missing data imputation techniques in future work to address this.
  • Focus on Complete Case Analysis: Related to the above, the reliance on complete case analysis is a limitation.
  • Interaction Effects: The authors suggest that future studies should consider the interaction between features (including social determinants and clinical variables) for a more comprehensive understanding.
  • Developing Versatile Systems: The primary future direction is the development of more versatile diagnostic systems that can account for and mitigate the impact of social determinant disparities, moving beyond a "one-size-fits-all" approach.
  • Investigating Underlying Causes: While the study identifies disparities, further research could delve into the underlying reasons for these performance drops, such as data representation bias or differential disease presentation across groups.

Summary updated

BibTeX
@article{wang2022comparison,
  author = {Hanyin Wang and Yikuan Li and Andrew Naidech and Yuan Luo},
  title = {{Comparison between machine learning methods for mortality prediction for sepsis patients with different social determinants}},
  journal = {BMC Medical Informatics Decis. Mak.},
  year = {2022},
  doi = {10.1186/s12911-022-01871-0},
}
Figure