Predicting Outcome of Endovascular Treatment for Acute Ischemic Stroke: Potential Value of Machine Learning Algorithms

Hendrikus J. A. van Os, Lucas A. Ramos, Adam Hilbert, Matthijs van Leeuwen, Marianne A.a. van Walderveen, Nyika D. Kruyt, Diederik W.j. Dippel, Ewout W. Steyerberg, Irene C. van der Schaaf, Hester F. Lingsma, Wouter J. Schonewille, Charles B.l.m. Majoie, Sílvia D. Olabarriaga, Koos Zwinderman, Esmée Venema, Henk A. Marquering, Marieke J.h. Wermer, the Mr Clean Registry InvestigatorsFrontiers in Neurology 2018 · DOI

TL;DR

This study investigated whether machine learning (ML) algorithms could outperform traditional logistic regression in predicting outcomes for acute ischemic stroke patients undergoing endovascular treatment (EVT). The findings suggest that while ML models showed moderate performance in predicting functional independence, they did not significantly outperform logistic regression, particularly when using only baseline patient data.

What Problem Does This Paper Address?

Acute ischemic stroke patients with large vessel occlusions (LVOs) can benefit significantly from endovascular treatment (EVT). However, accurately predicting an individual patient's outcome after EVT is crucial for personalized care and decision-making. Traditional statistical models, like logistic regression, are commonly used for this purpose. Machine learning, with its ability to capture complex, non-linear relationships and interactions between variables, is hypothesized to offer superior predictive power. This paper aims to rigorously compare the performance of various ML algorithms against logistic regression in predicting both radiological (reperfusion) and clinical (functional independence) outcomes after EVT, using a comprehensive dataset.

Methods

The researchers employed a comparative approach, evaluating several machine learning algorithms: Random Forests, Support Vector Machine (SVM), Neural Network, and Super Learner (an ensemble method). These were benchmarked against logistic regression models. To ensure a robust comparison, they utilized a nested cross-validation (CV) strategy. This involved an outer loop of 100 repeated random splits for training (80%) and testing (20%) to provide stable performance estimates, and an inner loop of 10-fold CV for hyperparameter tuning and variable selection for the logistic regression models. They explored two prediction settings: (1) using only baseline variables available at admission, and (2) using both baseline and treatment-related variables. Performance was primarily assessed using the Area Under the Receiver Operating Characteristic Curve (AUC).

Datasets

The study utilized data from the MR CLEAN Registry, an ongoing, prospective, observational monitoring study of patients undergoing EVT for acute ischemic stroke in the Netherlands. The dataset included 1,383 patients who met specific inclusion criteria (e.g., age ≥ 18, LVO in the anterior circulation). The data comprised a wide range of clinical baseline characteristics (e.g., NIHSS, GCS, medical history, medications) and radiological parameters (e.g., ASPECTS, clot burden, collaterals). Treatment-specific variables were also collected.

Key Insights and Ideas

  • ML vs. Traditional Models: The study directly addresses the ongoing debate about whether advanced ML techniques offer a significant advantage over well-established methods like logistic regression in clinical prediction tasks.
  • Data Availability Matters: The performance of predictive models was notably better when treatment-related variables were included, highlighting the importance of dynamic information beyond initial patient presentation.
  • Reperfusion is Hard to Predict: Predicting successful reperfusion (a key radiological outcome) was challenging for all models, even with extensive data. This suggests that factors beyond readily available clinical and imaging data at admission play a significant role.
  • Transparency vs. Performance Trade-off: While ML models can be powerful, their "black-box" nature can be a drawback in clinical settings where understanding why a prediction is made is important. Logistic regression, while potentially less powerful, offers greater interpretability.

Key Results

  • Reperfusion Prediction: All models, both ML and logistic regression, performed poorly in predicting good reperfusion, with mean AUCs ranging from 0.53 to 0.57, regardless of whether only baseline or baseline and treatment variables were used.
  • Functional Independence (Baseline Variables): For predicting 3-month functional independence using only baseline variables, models showed moderate performance, with mean AUCs ranging from 0.77 to 0.79. The best ML model (Super Learner) and the best logistic regression model (LASSO) had similar AUCs (0.79 vs. 0.78).
  • Functional Independence (Baseline + Treatment Variables): When including baseline and treatment variables, all models performed well, achieving mean AUCs between 0.88 and 0.91. Crucially, the difference in performance between the best ML algorithm (Random Forests, AUC 0.91) and the best logistic regression model (prior knowledge, AUC 0.90) was negligible (0.01).

Limitations and Future Work

The authors acknowledge several limitations:

  • Overfitting: ML models are prone to overfitting, which can lead to overly optimistic performance estimates. They mitigated this using nested cross-validation.
  • Small AUC Differences: While statistical significance was achieved due to the large sample size and numerous CV folds, the clinical relevance of the minimal AUC differences between models is questionable.
  • Registry Data: Registries can be subject to selection bias, although the authors argue that the MR CLEAN Registry likely minimized this by covering most EVT centers in the Netherlands.

Future work could explore:

  • The impact of more sophisticated variable selection methods for ML algorithms.
  • Investigating the specific interactions and non-linear relationships that ML models might capture, even if they didn't lead to superior performance in this specific dataset.
  • Developing more interpretable ML models to bridge the gap between predictive power and clinical understanding.
  • The authors also make their analysis pipeline and code publicly available, encouraging further research and replication.

Summary updated

Abstract

<b>Background:</b> Endovascular treatment (EVT) is effective for stroke patients with a large vessel occlusion (LVO) of the anterior circulation. To further improve personalized stroke care, it is essential to accurately predict outcome after EVT. Machine learning might outperform classical prediction methods as it is capable of addressing complex interactions and non-linear relations between variables. <b>Methods:</b> We included patients from the Multicenter Randomized Clinical Trial of Endovascular Treatment for Acute Ischemic Stroke in the Netherlands (MR CLEAN) Registry, an observational cohort of LVO patients treated with EVT. We applied the following machine learning algorithms: Random Forests, Support Vector Machine, Neural Network, and Super Learner and compared their predictive value with classic logistic regression models using various variable selection methodologies. Outcome variables were good reperfusion (post-mTICI ≥ 2b) and functional independence (modified Rankin Scale ≤2) at 3 months using (1) only baseline variables and (2) baseline and treatment variables. Area under the ROC-curves (AUC) and difference of mean AUC between the models were assessed. <b>Results:</b> We included 1,383 EVT patients, with good reperfusion in 531 (38%) and functional independence in 525 (38%) patients. Machine learning and logistic regression models all performed poorly in predicting good reperfusion (range mean AUC: 0.53-0.57), and moderately in predicting 3-months functional independence (range mean AUC: 0.77-0.79) using only baseline variables. All models performed well in predicting 3-months functional independence using both baseline and treatment variables (range mean AUC: 0.88-0.91) with a negligible difference of mean AUC (0.01; 95%CI: 0.00-0.01) between best performing machine learning algorithm (Random Forests) and best performing logistic regression model (based on prior knowledge). <b>Conclusion:</b> In patients with LVO machine learning algorithms did not outperform logistic regression models in predicting reperfusion and 3-months functional independence after endovascular treatment. For all models at time of admission radiological outcome was more difficult to predict than clinical outcome.
BibTeX
@inproceedings{os2018predicting,
  author = {Hendrikus J. A. van Os and Lucas A. Ramos and Adam Hilbert and Matthijs van Leeuwen and Marianne A.A. van Walderveen and Nyika D. Kruyt and Diederik W.J. Dippel and Ewout W. Steyerberg and Irene C. van der Schaaf and Hester F. Lingsma and Wouter J. Schonewille and Charles B.L.M. Majoie and Sílvia D. Olabarriaga and Koos Zwinderman and Esmée Venema and Henk A. Marquering and Marieke J.H. Wermer and the MR CLEAN Registry Investigators},
  title = {{Predicting Outcome of Endovascular Treatment for Acute Ischemic Stroke: Potential Value of Machine Learning Algorithms}},
  booktitle = {Frontiers in Neurology},
  year = {2018},
  doi = {10.3389/fneur.2018.00784},
}
Figure