CARDIAC-FM: A Multimodal Foundation Model for Cardiovascular Risk Prediction Using ECG and Cardiac MRI

Si Liu, Siting Li, Yi Qian, Bojun Chen, Jennifer A Brody, Vidhushei Yogeswaran, Kerri L Wiggins, Colleen M Sitlani, Jc Bis, Ali Shojaie, W T Longstreth, B. M. Psaty, Geoffrey H Tison, Simon S. Du, James S Floyd, Ting Ye2026 · DOI

TL;DR

This paper introduces CARDIAC-FM, a novel multimodal foundation model that leverages both ECG and cardiac MRI data to predict cardiovascular risk. By learning joint representations, CARDIAC-FM significantly improves prediction accuracy for conditions like atrial fibrillation and heart failure, even when only ECG data is available at inference, paving the way for more accessible and accurate cardiovascular risk assessment.

What Problem Does This Paper Address?

Cardiovascular diseases, particularly atrial fibrillation and heart failure, pose a significant global health burden. While existing risk prediction models exist, they often suffer from limited accuracy and poor generalizability across diverse populations. Traditional risk factors, while useful, don't capture the full picture of cardiac health. Furthermore, deep learning models have shown promise using single modalities like ECG or cardiac MRI, but integrating these complementary data sources effectively for robust, generalizable prediction has remained a challenge. The authors aim to bridge this gap by developing a multimodal foundation model that can learn from both ECG and MRI, leading to more accurate and widely applicable cardiovascular risk prediction.

Methods

The core of CARDIAC-FM is a two-stage multimodal foundation model trained using contrastive learning.

  • Stage 1: Multimodal Pre-training: This stage uses paired ECG and cardiac MRI data to train modality-specific encoders (an ECG Foundation Model and a CNN-LSTM for MRI). Through a symmetric InfoNCE contrastive loss, the model learns to align representations from both modalities in a shared latent space. This allows the ECG encoder to implicitly learn structural and functional information typically captured by MRI.
  • Stage 2: Downstream Prediction: A lightweight prediction head is added to the pre-trained encoders. The model is then fine-tuned using supervised learning for specific outcome predictions. This two-stage approach allows for efficient adaptation to various clinical endpoints without requiring extensive retraining from scratch for each new condition.

At inference, CARDIAC-FM supports flexible input configurations: ECG-only, ECG + MRI, ECG + clinical risk scores, and ECG + MRI + clinical risk scores, enhancing its applicability in real-world clinical settings with varying data availability.

Datasets

  • UK Biobank: Used for training the CARDIAC-FM model. It provided 57,609 paired ECG-cardiac MRI samples.
  • Cardiovascular Health Study (CHS): An external cohort used for validation.
  • Multi-Ethnic Study of Atherosclerosis (MESA): Another external cohort used for validation.

The study also evaluated predictions on a broad range of additional cardiovascular outcomes in CHS and MESA.

Key Insights and Ideas

  • Multimodal Pre-training Enhances Unimodal Performance: The most striking insight is that pre-training with both ECG and MRI significantly improves the performance of the ECG-only model. This suggests that learning from MRI enriches the ECG encoder's ability to capture crucial cardiac information, even when MRI is not available at inference.
  • ECG and Traditional Risk Factors are Complementary: Combining ECG features with established clinical risk scores consistently yielded additive gains in prediction accuracy, highlighting that these sources capture distinct but valuable dimensions of cardiovascular risk.
  • Foundation Models Enable Efficient Adaptation: The two-stage pre-training and fine-tuning framework allows CARDIAC-FM to adapt to new clinical endpoints with minimal task-specific fine-tuning, which is crucial for real-world scenarios where labeled data for specific diseases might be scarce.
  • CARDIAC-FM as a Cross-Modality Translator: The model can infer MRI-derived structural markers from ECG alone, enabling large-scale studies of cardiac structure and its relationship to outcomes in cohorts lacking imaging data.

Key Results

  • Superior Performance: CARDIAC-FM consistently outperformed unimodal AI models and traditional clinical risk scores across all evaluated cohorts (UK Biobank, CHS, MESA) for predicting atrial fibrillation and heart failure.
  • ECG + Risk Score Advantage: For atrial fibrillation prediction without MRI, CARDIAC-FM combining ECG with clinical risk scores achieved the highest discrimination (AUROC 0.82 in UK Biobank).
  • Heart Failure Gains: Adding MRI to CARDIAC-FM provided more substantial gains for heart failure prediction compared to atrial fibrillation. The full CARDIAC-FM model (ECG + MRI + Risk Scores) achieved AUROCs of 0.82 for atrial fibrillation and 0.83 for heart failure in UK Biobank.
  • Broad Applicability: CARDIAC-FM demonstrated strong generalizability across diverse populations and performed well on a wide range of cardiovascular outcomes (myocardial infarction, ischemic stroke, cardiovascular death, all-cause death) with minimal fine-tuning.
  • Risk Stratification: CARDIAC-FM improved risk stratification compared to clinical risk scores alone, particularly in the CHS cohort.

Limitations and Future Work

  • MRI Availability: While CARDIAC-FM can predict using ECG alone, its full potential is realized with both ECG and MRI. The authors acknowledge that MRI is expensive and not universally available, limiting its widespread clinical deployment.
  • Heart Failure Phenotypes: While CARDIAC-FM showed benefits for heart failure prediction, the authors note that future improvements in imaging encoders could further strengthen performance, particularly for specific heart failure subtypes.
  • Data Scarcity for Fine-tuning: The study highlights the effectiveness of minimal fine-tuning in data-scarce settings, but further research could explore optimal fine-tuning strategies for various clinical scenarios.
  • Future Improvements: The authors suggest that future work could focus on enhancing imaging encoders to further improve heart failure prediction. They also emphasize the promise of multimodal foundation modeling for integrating heterogeneous clinical data into predictive cardiovascular medicine.

Summary updated

Abstract

Abstract Atrial fibrillation and heart failure impose substantial health burdens worldwide, yet existing prediction models lack sufficient accuracy and generalizability. We developed CARDIAC-FM, a multimodal foundation model that learns joint representations of 12-lead electrocardiogram (ECG) and cardiac magnetic resonance imaging (MRI) through contrastive learning. We trained CARDIAC-FM on 57,609 paired ECG-cardiac MRI samples from UK Biobank and evaluated it in two external cohorts: the Cardiovascular Health Study (CHS) and the Multi-Ethnic Study of Atherosclerosis (MESA). CARDIAC-FM consistently outperformed unimodal models across all cohorts, and jointly incorporating ECG features with established clinical risk scores yielded additive gains in discrimination, indicating that ECG and traditional risk factors capture complementary dimensions of cardiovascular risk. The learned representations improved prediction across a range of cardiovascular outcomes with minimal task-specific fine-tuning, reflecting real-world settings where many diseases have limited positive samples and lack dedicated risk models. Although trained on paired ECG and MRI data, CARDIAC-FM generates predictions using ECG alone or ECG combined with established risk scores, enabling broad clinical deployment without MRI. These findings demonstrate the promise of multimodal pre-training for generalizable cardiovascular risk prediction.
BibTeX
@misc{liu2026cardiac,
  author = {Si Liu and Siting Li and Yi Qian and Bojun Chen and Jennifer A Brody and Vidhushei Yogeswaran and Kerri L Wiggins and Colleen M Sitlani and JC Bis and Ali Shojaie and W T Longstreth and B. M. Psaty and Geoffrey H Tison and Simon S. Du and James S Floyd and Ting Ye},
  title = {{CARDIAC-FM: A Multimodal Foundation Model for Cardiovascular Risk Prediction Using ECG and Cardiac MRI}},
  year = {2026},
  doi = {10.64898/2026.03.16.26348526},
}
Figure