TL;DR
This paper introduces FLAME, a machine learning model that predicts flares in patients with Systemic Lupus Erythematosus (SLE) by integrating electronic health records (EHRs) with social determinants of health (SDoH). The model is designed to be fair across different racial and ethnic groups, offering a more holistic and equitable approach to managing this chronic autoimmune disease.
What Problem Does This Paper Address?
Systemic Lupus Erythematosus (SLE) is a chronic autoimmune disease that disproportionately affects women and racial/ethnic minorities, often leading to significant health disparities. Predicting disease flares—periods of increased symptom severity—is crucial for improving patient outcomes and guiding treatment. However, existing prediction models often rely solely on clinical data and may not adequately account for the complex interplay of social and environmental factors (SDoH) that significantly influence SLE. Furthermore, many machine learning models in healthcare suffer from biases present in observational data, potentially exacerbating existing health inequities. This paper aims to address these challenges by developing a fair and explainable machine learning model that integrates both clinical and SDoH data to predict SLE flares.
Methods
The researchers developed FLAME (FLAre Machine learning prediction of SLE), a machine learning pipeline that incorporates a multi-step approach:
- Data Preprocessing: This involved imputing missing values and addressing data imbalance using techniques like random oversampling and undersampling.
- Machine Learning Modeling: Two models were trained: XGBoost (chosen for its effectiveness in tree-based modeling and robustness) and logistic regression (a widely used and interpretable linear model). Hyperparameters were optimized using grid search cross-validation. Regularization techniques (LASSO, Ridge, elastic net) were also applied to balance predictive performance and interpretability.
- Performance Assessment: Model performance was evaluated using metrics such as the Area Under the Receiver Operating Characteristic (AUROC), F1 Score, Precision, Sensitivity, and Specificity on an independent testing dataset.
- Explanation: To understand the model's predictions, they employed explainable AI techniques, specifically SHapley Additive exPlanations (SHAP) values, to identify key predictors. Causal structure learning using Mixed Graphical Models with PC-Stable (MGM-PC-Stable) was used to uncover causal relationships between variables.
- Fairness Assessment: The equality of opportunity metric was used to assess fairness, specifically by measuring the false-negative rate (FNR) across different racial and ethnic groups.
Datasets
The study utilized a retrospective cohort of 28,433 patients with SLE from the University of Florida Health (UF Health) Integrated Data Repository EHRs spanning from 2011 to 2022. This clinical data was linked with 675 contextual-level SDoH variables. The data was split into a modeling set (2011-2021) and an independent evaluation set (2022). The modeling set was further divided into training (70%), validation (10%), and testing (20%) sets, with 100 bootstrapped samples used for model robustness.
Key Insights and Ideas
- SDoH Matters, Even if Not Directly Predictive: While contextual-level SDoH alone had lower predictive power (AUROC of 0.54), their integration with clinical data improved the overall model. More importantly, causal analysis revealed that SDoH variables play a crucial role in pathways leading to flares, influencing clinical factors like headaches. This highlights the importance of a holistic view beyond direct predictive contribution.
- Symptom-Level Predictors as Early Warnings: Features like "headache," "organic brain syndrome," and "pyuria" emerged as significant predictors. These are often symptom-based and accessible in routine care, suggesting they could serve as valuable early warning signals for clinicians.
- Temporal Dynamics of Predictors: The Sankey diagram visualization revealed interesting temporal patterns for key predictors. For instance, "headache" and "organic brain syndrome" showed a consistent increase leading up to a flare and remained elevated afterward, while "pyuria" stabilized post-flare. This temporal understanding can inform more precise monitoring strategies.
- Achieving Algorithmic Fairness: The study successfully demonstrated that the FLAME model, even when incorporating SDoH, did not exhibit substantial disparities in predictive performance (measured by FNR) across racial and ethnic groups, a critical step towards equitable healthcare.
Key Results
- The FLAME model, integrating both clinical and contextual-level SDoH, achieved an AUROC of 0.66.
- The clinical-only model performed slightly better with an AUROC of 0.67.
- The SDoH-only model had a significantly lower AUROC of 0.54.
- Headache, organic brain syndrome, and pyuria were identified as the most significant predictors of flares.
- A contextual-level SDoH variable, "percentage of the population consisting of US citizens by naturalization," was also identified as important.
- Fairness assessments showed no significant biases in the false-negative rate across racial and ethnic groups.
- The top 10% of patients predicted by the FLAME model accounted for 59.88% of the risk of flares within a 3-month period.
Limitations and Future Work
- Diagnostic Accuracy: The cohort was primarily identified using ICD codes, which may lead to diagnostic inaccuracies. Future work will involve using Natural Language Processing (NLP) to analyze clinical notes for improved accuracy.
- Generalizability: While the cohort was diverse, it was drawn from a specific region (University of Florida Health). Generalizability to other regions and internationally needs further validation, potentially through federated learning and geographically diverse data sources.
- Individual-Level SDoH: The study used contextual-level SDoH (e.g., neighborhood data) but did not include individual-level SDoH (e.g., personal income, education, employment). Future research should incorporate individual-level SDoH to capture patient-specific stressors more effectively.
- Model Optimization: The selection and tuning of ML models followed standard practices, which might have limited the search space. Future efforts will explore automated ML pipelines for enhanced accuracy and reliability.
- Flare Definition: The study used a relative increase in SLEDAI-2K to define flares. Future studies could investigate alternative flare definitions and evaluate the impact of excluding overlapping predictors (e.g., symptoms that are also part of SLEDAI-2K) on model performance.
Summary updated