TL;DR
This paper introduces DAMS, a machine learning-based decision support tool designed to help neurologists identify mild stroke patients at high risk of post-stroke disability (PSD). By leveraging readily available clinical data, DAMS aims to improve individualized treatment decisions in emergency settings, with a faster version, R-DAMS, also developed for urgent situations.
What Problem Does This Paper Address?
Mild stroke, while often perceived as less severe, can still lead to significant post-stroke disability (PSD) in a substantial portion of patients. The optimal treatment for these individuals is not always clear-cut, presenting a dilemma for clinicians. Specifically, the decision to administer intravenous thrombolysis (IV alteplase) involves a trade-off between potential benefits and the risk of complications like intracranial hemorrhage. Current guidelines offer some direction, but the definition of a "disabling" stroke can be subjective, and existing risk prediction models often fail to accurately identify mild stroke patients who are at high risk of PSD, particularly those with an admission National Institute of Health Stroke Scale (NIHSS) score of 5 or less. This gap leaves a significant number of patients vulnerable to preventable disability.
Methods
The researchers developed a decision support tool called DAMS (Disability After Mild Stroke) using machine learning (ML) algorithms. They prospectively collected data from ischemic stroke patients treated at a major stroke center. Five ML models were trained and evaluated: logistic regression (LR), support vector machine (SVM), random forest classifier (RFC), extreme gradient boosting (XGB), and deep neural network (DNN).
Key methodological steps included:
- Data Preprocessing: Handling missing values using k-nearest neighbor imputation, standardizing continuous data, and one-hot encoding categorical data.
- Feature Selection: Employing the Least Absolute Shrinkage and Selection Operator (LASSO) algorithm to identify the most relevant variables, reducing overfitting and improving model interpretability.
- Model Training and Validation: Using an 80% training set and a 20% testing set, with 10-fold cross-validation during training to enhance generalizability. Hyperparameters were tuned using grid search to optimize the Area Under the Curve (AUC) of the Receiver Operating Characteristic (ROC).
- Model Evaluation: Assessing performance using AUC, sensitivity, specificity, accuracy, Brier score (for calibration), calibration slope, and calibration intercept. Decision curve analysis was used to evaluate clinical utility.
- Feature Importance: Utilizing the SHapley Additive exPlanations (SHAP) approach to understand which features contributed most to the model's predictions.
- Rapid Version Development: Creating a faster version, R-DAMS, by excluding variables that might take longer to obtain in an emergency setting (e.g., triglycerides, creatinine).
Datasets
The study utilized a prospective cohort of 1,905 ischemic stroke patients admitted to the National Advanced Stroke Center of Nanjing First Hospital (China) between July 2016 and September 2020. This dataset was split into training (80%) and testing (20%) sets for model development and evaluation. No new benchmark datasets were introduced; rather, the performance of the developed ML models was compared against existing scoring systems like THRIVE and HIAT.
Key Insights and Ideas
- Machine Learning for Mild Stroke Prediction: The study demonstrates the efficacy of ML in predicting post-stroke disability in mild stroke patients, a group often underserved by existing risk models.
- SVM as the Optimal Model: Among the tested ML algorithms, the Support Vector Machine (SVM) model for DAMS showed the best balance of discriminative performance and calibration, making it the chosen model.
- NIHSS is King (Still): Despite the complexity of ML models, the National Institutes of Health Stroke Scale (NIHSS) score on admission emerged as the single most important predictor of PSD, even within the narrow range of scores considered for mild stroke. This highlights the continued relevance of this established clinical tool.
- The Need for Speed: Recognizing that some crucial data points for DAMS might not be immediately available in an emergency, the development of R-DAMS offers a practical solution for faster decision-making.
- Clinical Utility Beyond Accuracy: While AUC is a standard metric, the study emphasizes the importance of calibration (how well predicted probabilities match actual outcomes) and clinical utility (decision curve analysis) for a truly useful decision support tool.
Key Results
- A total of 1,905 mild stroke patients were included, with 23.4% (447) experiencing post-stroke disability (PSD).
- The five ML models (LR, SVM, RFC, XGB, DNN) showed similar discriminative performance, with AUCs ranging from 0.691 to 0.823 on the training set and 0.749 to 0.766 on the testing set.
- The SVM model (selected for DAMS) achieved a Brier score of 0.159, a calibration slope of 0.935, and a calibration intercept of 0.035 on the testing set, indicating good calibration.
- The SHAP analysis identified NIHSS on admission, age, and fasting blood glucose (FBG) as the most crucial features for predicting PSD in DAMS.
- The rapid version, R-DAMS, showed comparable discriminative performance to DAMS but slightly worse calibration.
- The AUCs of the ML models were significantly better than those of traditional scoring systems (THRIVE and HIAT).
Limitations and Future Work
The authors acknowledge several limitations:
- External Validation: The study's findings are based on a single-center cohort, and external validation in diverse populations is crucial to assess generalizability.
- Selection Bias: As with any prediction model, inherent selection biases may affect transferability to other institutions.
- mRS Limitations: The modified Rankin Scale (mRS) used to assess PSD may not fully capture cognitive and mood outcomes, which can significantly impact a patient's social reintegration.
- Absence of Recurrent Stroke Data: Recurrent cerebrovascular events, a known predictor of PSD, were not included in the model development as this data is often unavailable in emergency contexts.
Future work should focus on:
- External Validation: Conducting studies in different centers to confirm the model's performance.
- Impact Studies: Quantifying how the application of DAMS and R-DAMS impacts neurologists' decision-making and patient outcomes.
- Software Development: Creating user-friendly software that provides clear interpretations of predictions and actionable treatment/prevention advice.
Summary updated