Stroke risk prediction using machine learning: a prospective cohort study of 0.5 million Chinese adults

Matthew Chun, Robert Clarke, Benjamin J. Cairns, David A. Clifton, Derrick Bennett, Yiping Chen, Yu Guo, Pei Pei, Jun Lv, Canqing Yu, Ling Yang, Liming Li, Zhengming Chen, Tingting Zhu

TL;DR

This study developed and evaluated machine learning (ML) models for stroke risk prediction in a large Chinese cohort, finding that gradient boosted trees (GBT) outperformed traditional Cox models. An ensemble approach combining Cox and GBT models further improved accuracy and specificity, offering a promising avenue for more effective clinical risk stratification.

What Problem Does This Paper Address?

Stroke is a major global health burden, particularly in low- and middle-income countries like China. Accurate prediction of stroke risk is crucial for implementing preventive measures. While traditional statistical models like Cox regression are widely used, their performance can be limited, especially in diverse contemporary populations. Existing risk prediction models, often developed in high-income countries, may not be directly applicable to Chinese populations due to differences in lifestyle, genetics, and healthcare. This paper addresses the need for improved stroke risk prediction models tailored to Chinese adults, exploring the potential of machine learning techniques to surpass traditional methods and developing a practical approach for clinical implementation.

Methods

The researchers compared several predictive modeling techniques, including traditional Cox regression and various machine learning algorithms: logistic regression, support vector machines, random survival forests, gradient boosted trees (GBT), and multilayer perceptrons. They trained and validated these models on a large prospective cohort dataset. A key innovation was the development of an ensemble model. This approach didn't simply average predictions; instead, it used a decision tree to intelligently select between the output of a Cox model and the best-performing ML model (GBT) for individuals where their predictions disagreed. This aimed to leverage the interpretability of Cox models while benefiting from the predictive power of ML where it offered an advantage. The models were evaluated on their discrimination (ability to distinguish between those who will and will not have a stroke) and calibration (how well predicted risks matched observed outcomes).

Datasets

The study utilized data from the China Kadoorie Biobank (CKB), a prospective cohort of 503,842 adults in China, recruited between 2004 and 2008. This dataset included sociodemographic factors, lifestyle information, medical history, physical activity, and physical measurements. The data was split into training (85%), validation (12.75%), and test (2.25%) sets for model development and evaluation. The performance of the developed models was benchmarked against the 2017 Framingham Stroke Risk Profile (FSRP).

Key Insights and Ideas

  • ML's Edge in Calibration: While discrimination improvements of ML over Cox models were marginal, the calibration improvements were substantial. This is critical for clinical practice, where precise risk thresholds (e.g., for initiating statin therapy) are used. Poor calibration can lead to misclassification of high-risk individuals.
  • GBT as a Strong Performer: Gradient Boosted Trees (GBT) emerged as the top-performing individual ML model, demonstrating superior discrimination and calibration compared to other ML algorithms and the traditional Cox model.
  • Ensemble for Practicality: The ensemble approach, which intelligently combined Cox and GBT, offered a practical way to integrate ML into clinical workflows. It prioritized the widely accepted Cox model for most individuals, only deferring to the ML model when there was significant disagreement, thus addressing potential clinician hesitancy towards purely ML-based tools.
  • Time-Dependent Performance: Binary classification ML models (retrained for specific intervals) showed better performance for shorter follow-up periods (0-3, 3-6, 6-9 years) compared to survival models, suggesting their utility for more immediate risk assessments.

Key Results

  • The Gradient Boosted Tree (GBT) model achieved the best discrimination (AUROC: 0.833 in men, 0.836 in women) and calibration for 9-year stroke risk prediction.
  • All developed models significantly outperformed the original Framingham Stroke Risk Profile (FSRP), which tended to underestimate stroke risk in the CKB cohort.
  • The ensemble approach, combining Cox and GBT, yielded incremental improvements in accuracy (76% in men, 80% in women) and specificity (76% in men, 81% in women) compared to single-model approaches.
  • The ensemble approach correctly identified high-risk individuals with a positive predictive value (PPV) of 26% in men and 24% in women.

Limitations and Future Work

The authors acknowledge several limitations:

  • Absence of Atrial Fibrillation (AF) Data: Atrial fibrillation, a known stroke risk factor, was not recorded in the CKB and thus could not be included in the models. However, they argue its prevalence is lower in China, minimizing its impact.
  • Exclusion of Right-Censored Data for Binary Models: Binary classification models excluded individuals who were lost to follow-up or died before the prediction interval, which is an inherent limitation of this modeling approach.
  • Generalizability: The proposed risk equations were not designed for immediate implementation and require validation in independent Chinese populations and potentially other LMICs, as the CKB cohort may not be fully representative.
  • Cost-Benefit Analysis: Further work is needed to compare the cost and benefits of implementing these ML-based approaches versus existing care guidelines.

Future work should focus on external validation, cost-effectiveness studies, and exploring the integration of these models into clinical practice to improve stroke prevention.

Summary updated

BibTeX
@article{chun2021stroke,
  author = {Matthew Chun and Robert Clarke and Benjamin J. Cairns and David A. Clifton and Derrick Bennett and Yiping Chen and Yu Guo and Pei Pei and Jun Lv and Canqing Yu and Ling Yang and Liming Li and Zhengming Chen and Tingting Zhu},
  title = {{Stroke risk prediction using machine learning: a prospective cohort study of 0.5 million Chinese adults}},
  journal = {J. Am. Medical Informatics Assoc.},
  year = {2021},
  doi = {10.1093/jamia/ocab068},
}
Figure