TL;DR
Researchers have developed a multidimensional machine learning framework using Electronic Health Records (EHR) to retrospectively identify suicide-related risk. By combining a novel age-matched resampling strategy with a calibrated XGBoost model, the team achieved an F1 score of 0.81, highlighting that diagnostic complexity and social determinants are just as critical as psychiatric history in assessing risk.
What Problem Does This Paper Address?
Suicide prevention is a massive public health challenge, with nearly 50,000 deaths in the U.S. in 2023 alone. While EHRs contain a treasure trove of clinical and behavioral data, building predictive models from them is notoriously difficult due to two main issues:
- Severe Class Imbalance: Suicide-related events are (thankfully) rare in the general population, often making up less than 5% of a dataset. Standard models often struggle to learn from such "needle-in-a-haystack" scenarios.
- Age-Related Confounding: Health profiles and suicide prevalence change drastically with age. Without careful control, a model might inadvertently learn to use age as a proxy for risk rather than actual clinical indicators.
The authors aim to create a robust, interpretable framework that integrates psychiatric, medical, and social factors to better characterize high-risk patients.
Methods
The authors proposed a three-pronged approach:
- Feature Engineering: They mapped over 27,000 unique clinical diagnoses into 53 clinically motivated categories (shown in Fig. 1). These span psychiatric conditions, substance abuse, and social determinants of health (SDoH).
- Age-Matched Resampling: To combat class imbalance and confounding, they implemented a 3:1 matching strategy. For every patient with a suicide-related outcome, they sampled three "control" patients from the same age group. This ensured a consistent 25% positive rate across the analytical cohort.
- Model Pipeline: They evaluated Logistic Regression, Random Forests, and XGBoost. Crucially, they applied isotonic calibration to the models. This step aligns the model's predicted probabilities with actual observed frequencies, which is vital for clinical utility.
Datasets
The study utilized MIMIC-IV, a massive, de-identified EHR dataset containing records from over 200,000 patients admitted to the emergency department or intensive care units at Beth Israel Deaconess Medical Center. The final dataset included over 6.3 million recorded diagnoses.
Key Insights and Ideas
- The "Diagnostic Burden" Proxy: One of the most significant predictors was simply the total count of a patient's diagnoses. This suggests that overall medical complexity and frequent healthcare utilization are strong indicators of underlying risk.
- SDoH are High-Impact: While depression is a known factor, social determinants like exposure to violence and housing instability showed massive univariate associations (Odds Ratios of 34.6 and 16.6, respectively).
- Calibration is Essential: Raw machine learning scores often over- or under-estimate risk. The authors showed that isotonic calibration significantly improved the reliability of predicted probabilities, making the model's output more "honest."
Key Results
- Top Performer: The calibrated XGBoost model was the clear winner, achieving an F1 score of 0.81 and an Area Under the Precision-Recall Curve (AUPRC) of 0.89 on the age-matched cohort.
- Feature Importance: SHAP (Shapley Additive Explanations) analysis revealed that while "has_Depression" was the most influential feature, personality disorders, SDoH-related violence, and even specific age indicators (like the 76-85 bracket) played major roles in the model's decisions.
- Robustness: The model maintained stable performance across 20 different random resampling seeds, suggesting the findings aren't just a fluke of the specific patients chosen for the control group.
Limitations and Future Work
- Retrospective Bias: The study relies on documented ICD codes. Patients who struggle with suicidal ideation but never receive a formal diagnosis code for it would be misclassified as "low risk" in this training set.
- Elderly Performance Gap: The model's recall dropped significantly for patients over 76. The authors attribute this to smaller sample sizes in older cohorts and suggest that geriatric suicide risk may require specialized modeling.
- Next Steps: The authors plan to move from static "snapshot" modeling to temporal sequence modeling, capturing how a patient's risk evolves over time across multiple hospital visits.
Summary updated