StrokeClassifier: Ischemic Stroke Etiology Classification by Ensemble Consensus Modeling Using Electronic Health Records

Ho‐Joon Lee, Lee H. Schwamm, Lauren Sansing, Hooman Kamel, Adam de Havenon, Ashby C. Turner, Kevin N. Sheth, Smita Krishnaswamy, Cynthia Brandt, Hongyu Zhao, Harlan M. Krumholz, Richa Sharma

Introduction

Determining the etiology of an acute ischemic stroke (AIS) is a critical step in secondary stroke prevention. However, identifying the precise cause—whether cardioembolism, large artery atherosclerosis, small vessel disease, or other determined causes—remains diagnostically challenging. In the United States, approximately 20–30% of patients are discharged with a diagnosis of "cryptogenic" stroke, where the underlying cause remains unknown. This diagnostic gap often stems from the complexity of synthesizing vast amounts of imaging, laboratory, and clinical data under significant time pressure, compounded by a shortage of specialized vascular neurologists.

To address this, researchers developed StrokeClassifier, an automated machine intelligence tool designed to classify stroke etiology using electronic health record (EHR) text data. By leveraging natural language processing (NLP) and ensemble modeling, the tool aims to standardize the diagnostic process and provide expert-level classification at scale.

Methods and Study Design

The development of StrokeClassifier involved a multi-stage process utilizing data from 2,039 non-cryptogenic AIS patients across two major academic hospitals (Yale New Haven Hospital and Massachusetts General Hospital) between 2015 and 2020.

Overview of the StrokeClassifier development workflow, including feature extraction, ensemble modeling, and validation.

Feature Extraction and Engineering

The core of the system relies on unstructured text from discharge summaries. The researchers employed MetaMap, an NLP tool from the National Library of Medicine, to extract over 2,000 clinical concepts (Concept Unique Identifiers, or CUIs). This was supplemented by features extracted via regular expressions, covering social history, NIH Stroke Scale scores, radiology findings, and laboratory results.

Ensemble Modeling

StrokeClassifier is an ensemble consensus meta-model comprising nine different machine learning classifiers, including XGBoost, Random Forests, and Support Vector Classifiers. To ensure diagnostic reliability, the team implemented a "certainty heuristic": a diagnosis is deemed "confident" only when a high degree of consensus is reached among the individual models in the ensemble.

Validation

The model was externally validated using 406 discharge summaries from the publicly available MIMIC-III dataset (Beth Israel Deaconess Medical Center). Additionally, a "cryptogenic cohort" of 788 patients was used to evaluate the model's ability to identify hidden patterns in cases where human experts had not reached a definitive conclusion.

Key Insights and Results

The study yielded several significant findings regarding the utility of AI in clinical stroke management:

  1. The Value of Unstructured Data: Features extracted from clinical narratives (CUIs) were the strongest predictors of stroke etiology, outperforming traditional structured data. This suggests that the "narrative" recorded by clinicians contains vital diagnostic clues often missed by standard databases.
  2. Performance and Accuracy: StrokeClassifier achieved a mean cross-validated accuracy of 0.74 (±0.01) and a weighted F1 score of 0.74. In the external MIMIC-III cohort, it maintained an accuracy of 0.70 and a weighted F1 of 0.71, demonstrating robust performance across different EHR systems.
  3. Reducing Cryptogenic Diagnoses: When applied to the cryptogenic cohort using the certainty heuristic, the model successfully "unmasked" likely etiologies for a majority of cases. This reduced the overall percentage of cryptogenic strokes in the study population from 25.2% to just 7.2%. Many of these previously unknown cases showed signatures of cardioembolism, a finding that could fundamentally alter secondary prevention strategies (e.g., transitioning from antiplatelet therapy to anticoagulation).
  4. Explainability: SHapley Additive exPlanation (SHAP) analysis confirmed that the model's decisions were biologically plausible. The top features contributing to predictions included atrial fibrillation, patient age, middle cerebral artery occlusion, internal carotid artery occlusion, and frontal stroke location.

Discussion and Future Directions

StrokeClassifier represents a validated AI tool that rivals the performance of vascular neurologists in classifying ischemic stroke etiology. By mimicking the consensus of expert neurologists, it offers a scalable solution for clinical decision support.

However, the study has limitations. It is currently cross-sectional, focusing on a snapshot of the patient's record at discharge. Future work will involve applying the model to longitudinal data to determine if it can predict eventual diagnoses for patients whose stroke cause is only discovered months later. Furthermore, while the model performed well in academic centers, additional testing in community hospital settings is necessary to account for variations in documentation styles and diagnostic resource availability.

Ultimately, StrokeClassifier demonstrates that machine learning can bridge the diagnostic gap in stroke care, potentially leading to more personalized and effective secondary prevention for thousands of patients.

Summary updated

Abstract

Determining the etiology of an acute ischemic stroke (AIS) is fundamental to secondary stroke prevention efforts but can be diagnostically challenging. We trained and validated an automated classification machine intelligence tool, <i>StrokeClassifier</i>, using electronic health record (EHR) text data from 2,039 non-cryptogenic AIS patients at 2 academic hospitals to predict the 4-level outcome of stroke etiology determined by agreement of at least 2 board-certified vascular neurologists' review of the stroke hospitalization EHR. <i>StrokeClassifier</i> is an ensemble consensus meta-model of 9 machine learning classifiers applied to features extracted from discharge summary texts by natural language processing. <i>StrokeClassifier</i> was externally validated in 406 discharge summaries from the MIMIC-III dataset reviewed by a vascular neurologist to ascertain stroke etiology. Compared with stroke etiologies adjudicated by vascular neurologists, <i>StrokeClassifier</i> achieved the mean cross-validated accuracy of 0.74 (±0.01) and weighted F1 of 0.74 (±0.01). In the MIMIC-III cohort, the accuracy and weighted F1 of <i>StrokeClassifier</i> were 0.70 and 0.71, respectively. SHapley Additive exPlanation analysis elucidated that the top 5 features contributing to stroke etiology prediction were atrial fibrillation, age, middle cerebral artery occlusion, internal carotid artery occlusion, and frontal stroke location. We then designed a certainty heuristic to deem a <i>StrokeClassifier</i> diagnosis as confidently non-cryptogenic by the degree of consensus among the 9 classifiers, and applied it to 788 cryptogenic patients. This reduced the percentage of the cryptogenic strokes from 25.2% to 7.2% of all ischemic strokes. <i>StrokeClassifier</i> is a validated artificial intelligence tool that rivals the performance of vascular neurologists in classifying ischemic stroke etiology for individual patients. With further training, <i>StrokeClassifier</i> may have downstream applications including its use as a clinical decision support system.
BibTeX
@article{lee2024strokeclassifier,
  author = {Ho‐Joon Lee and Lee H. Schwamm and Lauren Sansing and Hooman Kamel and Adam de Havenon and Ashby C. Turner and Kevin N. Sheth and Smita Krishnaswamy and Cynthia Brandt and Hongyu Zhao and Harlan M. Krumholz and Richa Sharma},
  title = {{StrokeClassifier: Ischemic Stroke Etiology Classification by Ensemble Consensus Modeling Using Electronic Health Records}},
  journal = {Npj Digit. Medicine},
  year = {2024},
  doi = {10.1038/s41746-024-01120-w},
}
Figure