StrokeClassifier: ischemic stroke etiology classification by ensemble consensus modeling using electronic health records

Ho-Joon Lee, Lee H. Schwamm, Lauren H. Sansing, Hooman Kamel, Adam DeHavenon, Ashby C. Turner, Kevin N. Sheth, Smita Krishnaswamy, Cynthia Brandt, Hongyu Zhao, Harlan M. Krumholz, Richa Sharma

Abstract

Identifying the underlying cause (etiology) of an ischemic stroke is critical for preventing recurrence, yet 20–30% of cases remain "cryptogenic" or unknown. StrokeClassifier is an ensemble machine learning tool that utilizes natural language processing (NLP) to analyze electronic health record (EHR) text. By employing a consensus-based "certainty heuristic," the system achieves diagnostic accuracy comparable to vascular neurologists and successfully reclassified over 70% of cryptogenic strokes into specific, treatable categories.

Introduction

When a patient suffers an ischemic stroke, determining the etiology—such as a blood clot from the heart (cardioembolism) or a narrowed artery (large artery atherosclerosis)—is the most critical step for secondary prevention. The correct diagnosis dictates whether a patient receives specific blood thinners or undergoes surgery. However, synthesizing complex imaging, lab results, and clinical history is challenging. In the U.S., a significant portion of strokes are labeled "cryptogenic" due to incomplete workups or lack of specialist access. StrokeClassifier was developed to standardize this diagnostic process by extracting insights from unstructured clinical notes.

Methods

The researchers developed StrokeClassifier, an ensemble consensus meta-model. The pipeline involves several sophisticated stages of data processing and modeling.

The StrokeClassifier pipeline, from data extraction to model testing.
  1. Feature Extraction (NLP): Using MetaMap, the system scans discharge summaries to extract "Concept Unique Identifiers" (CUIs), converting unstructured clinical prose into standardized medical concepts like "atrial fibrillation."
  2. Multimodal Data Integration: The model combines these text-derived features with structured data, including age, sex, and radiologic findings such as infarct location and vessel patency.
  3. Ensemble Modeling: Instead of a single algorithm, the system trains 9 different classifiers (e.g., Logistic Regression, Random Forests, XGBoost).
  4. Certainty Heuristic: This core component measures agreement among the 9 internal classifiers. High confidence is assigned if 7 or more models agree on a cause; otherwise, the case is flagged as truly uncertain.

Datasets

The model was developed and validated using three distinct cohorts:

  • Derivation Cohort: 2,039 non-cryptogenic patients from Yale New Haven Hospital (YNHH) and Massachusetts General Hospital (MGH), with diagnoses adjudicated by board-certified vascular neurologists.
  • External Validation: 406 discharge summaries from the MIMIC-III dataset (Beth Israel Deaconess Medical Center).
  • Application Set: 788 patients originally diagnosed as "cryptogenic" to test the model's ability to identify missed causes.

Key Insights and Results

StrokeClassifier demonstrated substantial agreement with human specialists. A key finding was that the AI prioritized features—such as atrial fibrillation and frontal stroke location—that align with the clinical logic used by experts.

Prevalence of key clinical features across TOAST categories.
  • Performance Metrics: The model achieved a mean cross-validated accuracy of 0.74 and a weighted F1-score of 0.74. In binary tasks (e.g., identifying cardioembolic vs. others), accuracy ranged from 0.77 to 0.96.
  • Generalizability: The model maintained an accuracy of 0.70 on the external MIMIC-III dataset, indicating it is not overfitted to specific institutional documentation styles.
  • Reclassifying "Cryptogenic" Strokes: By applying the certainty heuristic, the model reduced the cryptogenic rate from 25.2% to 7.2%. This suggests that many "unknown" strokes have sufficient evidence in their charts for classification if reviewed systematically.

Discussion and Future Work

While StrokeClassifier shows great promise, it currently relies on discharge summaries written after the diagnostic workup. Future iterations aim to utilize "day zero" notes to assist real-time clinical decision-making. Additionally, while the model was tested across major academic centers, further work is needed to ensure performance in community hospitals with potentially less detailed documentation. The ultimate goal is to integrate StrokeClassifier into EHR systems as a real-time clinical decision support tool, guiding clinicians toward optimal preventive therapies.

Summary updated

Abstract

Determining the etiology of an acute ischemic stroke (AIS) is fundamental to secondary stroke prevention efforts but can be diagnostically challenging. We trained and validated an automated classification machine intelligence tool, <i>StrokeClassifier</i>, using electronic health record (EHR) text data from 2,039 non-cryptogenic AIS patients at 2 academic hospitals to predict the 4-level outcome of stroke etiology determined by agreement of at least 2 board-certified vascular neurologists' review of the stroke hospitalization EHR. <i>StrokeClassifier</i> is an ensemble consensus meta-model of 9 machine learning classifiers applied to features extracted from discharge summary texts by natural language processing. <i>StrokeClassifier</i> was externally validated in 406 discharge summaries from the MIMIC-III dataset reviewed by a vascular neurologist to ascertain stroke etiology. Compared with stroke etiologies adjudicated by vascular neurologists, <i>StrokeClassifier</i> achieved the mean cross-validated accuracy of 0.74 (±0.01) and weighted F1 of 0.74 (±0.01). In the MIMIC-III cohort, the accuracy and weighted F1 of <i>StrokeClassifier</i> were 0.70 and 0.71, respectively. SHapley Additive exPlanation analysis elucidated that the top 5 features contributing to stroke etiology prediction were atrial fibrillation, age, middle cerebral artery occlusion, internal carotid artery occlusion, and frontal stroke location. We then designed a certainty heuristic to deem a <i>StrokeClassifier</i> diagnosis as confidently non-cryptogenic by the degree of consensus among the 9 classifiers, and applied it to 788 cryptogenic patients. This reduced the percentage of the cryptogenic strokes from 25.2% to 7.2% of all ischemic strokes. <i>StrokeClassifier</i> is a validated artificial intelligence tool that rivals the performance of vascular neurologists in classifying ischemic stroke etiology for individual patients. With further training, <i>StrokeClassifier</i> may have downstream applications including its use as a clinical decision support system.
BibTeX
@article{lee2024strokeclassifier,
  author = {Ho-Joon Lee and Lee H. Schwamm and Lauren H. Sansing and Hooman Kamel and Adam DeHavenon and Ashby C. Turner and Kevin N. Sheth and Smita Krishnaswamy and Cynthia Brandt and Hongyu Zhao and Harlan M. Krumholz and Richa Sharma},
  title = {{StrokeClassifier: ischemic stroke etiology classification by ensemble consensus modeling using electronic health records}},
  journal = {Npj Digit. Medicine},
  year = {2024},
  doi = {10.1038/s41746-024-01120-w},
}
Figure