Abstract
Identifying the underlying cause (etiology) of an ischemic stroke is critical for preventing recurrence, yet 20–30% of cases remain "cryptogenic" or unknown. StrokeClassifier is an ensemble machine learning tool that utilizes natural language processing (NLP) to analyze electronic health record (EHR) text. By employing a consensus-based "certainty heuristic," the system achieves diagnostic accuracy comparable to vascular neurologists and successfully reclassified over 70% of cryptogenic strokes into specific, treatable categories.
Introduction
When a patient suffers an ischemic stroke, determining the etiology—such as a blood clot from the heart (cardioembolism) or a narrowed artery (large artery atherosclerosis)—is the most critical step for secondary prevention. The correct diagnosis dictates whether a patient receives specific blood thinners or undergoes surgery. However, synthesizing complex imaging, lab results, and clinical history is challenging. In the U.S., a significant portion of strokes are labeled "cryptogenic" due to incomplete workups or lack of specialist access. StrokeClassifier was developed to standardize this diagnostic process by extracting insights from unstructured clinical notes.
Methods
The researchers developed StrokeClassifier, an ensemble consensus meta-model. The pipeline involves several sophisticated stages of data processing and modeling.
- Feature Extraction (NLP): Using MetaMap, the system scans discharge summaries to extract "Concept Unique Identifiers" (CUIs), converting unstructured clinical prose into standardized medical concepts like "atrial fibrillation."
- Multimodal Data Integration: The model combines these text-derived features with structured data, including age, sex, and radiologic findings such as infarct location and vessel patency.
- Ensemble Modeling: Instead of a single algorithm, the system trains 9 different classifiers (e.g., Logistic Regression, Random Forests, XGBoost).
- Certainty Heuristic: This core component measures agreement among the 9 internal classifiers. High confidence is assigned if 7 or more models agree on a cause; otherwise, the case is flagged as truly uncertain.
Datasets
The model was developed and validated using three distinct cohorts:
- Derivation Cohort: 2,039 non-cryptogenic patients from Yale New Haven Hospital (YNHH) and Massachusetts General Hospital (MGH), with diagnoses adjudicated by board-certified vascular neurologists.
- External Validation: 406 discharge summaries from the MIMIC-III dataset (Beth Israel Deaconess Medical Center).
- Application Set: 788 patients originally diagnosed as "cryptogenic" to test the model's ability to identify missed causes.
Key Insights and Results
StrokeClassifier demonstrated substantial agreement with human specialists. A key finding was that the AI prioritized features—such as atrial fibrillation and frontal stroke location—that align with the clinical logic used by experts.
- Performance Metrics: The model achieved a mean cross-validated accuracy of 0.74 and a weighted F1-score of 0.74. In binary tasks (e.g., identifying cardioembolic vs. others), accuracy ranged from 0.77 to 0.96.
- Generalizability: The model maintained an accuracy of 0.70 on the external MIMIC-III dataset, indicating it is not overfitted to specific institutional documentation styles.
- Reclassifying "Cryptogenic" Strokes: By applying the certainty heuristic, the model reduced the cryptogenic rate from 25.2% to 7.2%. This suggests that many "unknown" strokes have sufficient evidence in their charts for classification if reviewed systematically.
Discussion and Future Work
While StrokeClassifier shows great promise, it currently relies on discharge summaries written after the diagnostic workup. Future iterations aim to utilize "day zero" notes to assist real-time clinical decision-making. Additionally, while the model was tested across major academic centers, further work is needed to ensure performance in community hospitals with potentially less detailed documentation. The ultimate goal is to integrate StrokeClassifier into EHR systems as a real-time clinical decision support tool, guiding clinicians toward optimal preventive therapies.
Summary updated