Many Models, Little Adoption—What Accounts for Low Uptake of Machine Learning Models for Atrial Fibrillation Prediction and Detection?

Yuki I. Kawamura, Alireza Vafaei Sadr, Vida Abedi, Ramin ZandJournal of Clinical Medicine 2024 · DOI

TL;DR

Machine learning (ML) models show great promise for predicting and detecting atrial fibrillation (AF), a major stroke risk factor, but their real-world adoption is severely limited. This review identifies key barriers to clinical translation, including poor generalizability, lack of transparency, and insufficient validation, and proposes actionable recommendations to bridge this gap.

What Problem Does This Paper Address?

Atrial fibrillation (AF) is a significant contributor to stroke, yet it often goes undiagnosed, particularly in its paroxysmal form. This diagnostic gap is critical because early detection and management, often through anticoagulation, can prevent devastating strokes. While machine learning (ML) offers powerful tools for identifying individuals at risk of developing AF or detecting it in stroke patients, the clinical translation of these ML models has been remarkably slow. This paper aims to systematically investigate why so many promising ML models for AF prediction and detection are not making their way into clinical practice, exploring the reasons behind this low adoption rate.

Methods

The authors conducted a scoping review of existing literature, searching major databases (MEDLINE, Embase, Web of Science, Clinicaltrials.gov, ICTRP) for studies published up to September 2022. They focused on peer-reviewed English articles that utilized ML methods for either predicting incident AF in individuals without a prior AF diagnosis or detecting AF in patients who had previously experienced a stroke. The review included studies that reported adequate performance metrics. After an initial search yielding 2815 articles, 16 studies on AF prediction and three on AF detection in stroke cohorts were selected for in-depth analysis. The authors then meticulously examined these selected studies for various aspects, including model architecture, input data, data preprocessing, performance metrics, validation strategies, and importantly, the limitations that might hinder clinical adoption.

Datasets

The paper does not introduce any new datasets. Instead, it analyzes studies that utilized various existing datasets, often derived from electronic health records (EHRs) or clinical trials. The specific datasets used by the individual studies are detailed in supplementary tables, but a centralized benchmark dataset for AF prediction or detection is not presented. The authors note that data formats and curation varied significantly across studies, contributing to the lack of generalizability.

Key Insights and Ideas

  • The "Many Models, Little Adoption" Paradox: Despite a surge in ML models for AF, very few have transitioned into clinical use, highlighting a significant disconnect between research and practice.
  • Data Limitations are Pervasive: Many models rely on a restricted subset of patient data, neglecting valuable information from EHRs, imaging, or biomarkers, which could lead to more holistic and accurate predictions.
  • External Validation is Crucial, Yet Scarce: A staggering majority of studies (63%) relied solely on internal validation, meaning models were tested on data from the same source they were trained on. This severely limits their generalizability to new patient populations or different clinical settings.
  • Transparency is Lacking: Crucially, none of the reviewed studies made their code or models publicly available. This lack of transparency hinders reproducibility, trust, and the ability for others to build upon or verify their findings.
  • Bias Mitigation is Overlooked: Data preprocessing often lacked explicit steps to address potential biases (e.g., related to race, socioeconomic status), raising concerns that ML models might perpetuate or even amplify existing healthcare disparities.

Key Results

The review identified 19 studies in total (16 for prediction, 3 for detection). Key findings regarding the limitations of these studies include:

  • Limited Data Utilization: Many models used only a small fraction of available patient health record variables.
  • Insufficient External Validation: Only 37% of models underwent external validation.
  • Lack of Transparency: 0% of models and only 53% of datasets were made publicly available, severely impacting reproducibility.
  • Bias Concerns: Data preprocessing often did not include bias mitigation strategies.
  • Performance Metrics: While ML models often outperformed traditional risk scores, issues like low generalizability, high false alarm rates, and lack of interpretability were frequently cited as barriers. For AF detection in stroke cohorts, sensitivity ranged from 63-95% and specificity from 35-96%, with positive predictive values (PPVs) as low as 23%.

Limitations and Future Work

The authors acknowledge several limitations:

  • Limited Studies for AF Detection in Stroke: While many AF prediction models exist, fewer studies specifically focused on detecting AF in stroke patients. Future work should externally validate existing algorithms on stroke cohorts.
  • Inability for Direct Model Comparison: Significant variations in model types, input data, and evaluation metrics made direct quantitative comparison of models impossible. Future studies should aim to compare multiple models on identical, unseen datasets.

The paper strongly recommends future work to address these limitations:

  • Improve Generalizability: Incorporate multimodal data, conduct robust external validation, and explore decentralized training methods (e.g., federated learning).
  • Enhance Transparency: Mandate the availability of source code and de-identified data for reproducibility and trust.
  • Mitigate Bias: Prioritize diverse training cohorts, implement fairness audits, and use post-processing techniques to address potential biases.
  • Develop Clear Clinical and Legal Frameworks: Address physician liability and patient education to facilitate smoother integration into clinical workflows.
  • Investigate Cost-Effectiveness: Further research is needed to determine the optimal target populations and screening strategies for cost-effective implementation.

Summary updated

Abstract

(1) <b>Background:</b> Atrial fibrillation (AF) is a major risk factor for stroke and is often underdiagnosed, despite being present in 13-26% of ischemic stroke patients. Recently, a significant number of machine learning (ML)-based models have been proposed for AF prediction and detection for primary and secondary stroke prevention. However, clinical translation of these technological innovations to close the AF care gap has been scant. Herein, we sought to systematically examine studies, employing ML models to predict incident AF in a population without prior AF or to detect paroxysmal AF in stroke cohorts to identify key reasons for the lack of translation into the clinical workflow. We conclude with a set of recommendations to improve the clinical translatability of ML-based models for AF. (2) <b>Methods</b>: MEDLINE, Embase, Web of Science, Clinicaltrials.gov, and ICTRP databases were searched for relevant articles from the inception of the databases up to September 2022 to identify peer-reviewed articles in English that used ML methods to predict incident AF or detect AF after stroke and reported adequate performance metrics. The search yielded 2815 articles, of which 16 studies using ML models to predict incident AF and three studies focusing on ML models to detect AF post-stroke were included. (3) <b>Conclusions</b>: This study highlights that (1) many models utilized only a limited subset of variables available from patients' health records; (2) only 37% of models were externally validated, and stratified analysis was often lacking; (3) 0% of models and 53% of datasets were explicitly made available, limiting reproducibility and transparency; and (4) data pre-processing did not include bias mitigation and sufficient details, leading to potential selection bias. Low generalizability, high false alarm rate, and lack of interpretability were identified as additional factors to be addressed before ML models can be widely deployed in the clinical care setting. Given these limitations, our recommendations to improve the uptake of ML models for better AF outcomes include improving generalizability, reducing potential systemic biases, and investing in external validation studies whilst developing a transparent modeling pipeline to ensure reproducibility.
BibTeX
@article{kawamura2024many,
  author = {Yuki I. Kawamura and Alireza Vafaei Sadr and Vida Abedi and Ramin Zand},
  title = {{Many Models, Little Adoption—What Accounts for Low Uptake of Machine Learning Models for Atrial Fibrillation Prediction and Detection?}},
  journal = {Journal of Clinical Medicine},
  year = {2024},
  doi = {10.3390/jcm13051313},
}
Figure