TL;DR
This paper introduces CARDIAC-FM, a novel multimodal foundation model that leverages both ECG and cardiac MRI data to predict cardiovascular risk. By learning joint representations, CARDIAC-FM significantly improves prediction accuracy for conditions like atrial fibrillation and heart failure, even when only ECG data is available at inference, paving the way for more accessible and accurate cardiovascular risk assessment.
What Problem Does This Paper Address?
Cardiovascular diseases, particularly atrial fibrillation and heart failure, pose a significant global health burden. While existing risk prediction models exist, they often suffer from limited accuracy and poor generalizability across diverse populations. Traditional risk factors, while useful, don't capture the full picture of cardiac health. Furthermore, deep learning models have shown promise using single modalities like ECG or cardiac MRI, but integrating these complementary data sources effectively for robust, generalizable prediction has remained a challenge. The authors aim to bridge this gap by developing a multimodal foundation model that can learn from both ECG and MRI, leading to more accurate and widely applicable cardiovascular risk prediction.
Methods
The core of CARDIAC-FM is a two-stage multimodal foundation model trained using contrastive learning.
- Stage 1: Multimodal Pre-training: This stage uses paired ECG and cardiac MRI data to train modality-specific encoders (an ECG Foundation Model and a CNN-LSTM for MRI). Through a symmetric InfoNCE contrastive loss, the model learns to align representations from both modalities in a shared latent space. This allows the ECG encoder to implicitly learn structural and functional information typically captured by MRI.
- Stage 2: Downstream Prediction: A lightweight prediction head is added to the pre-trained encoders. The model is then fine-tuned using supervised learning for specific outcome predictions. This two-stage approach allows for efficient adaptation to various clinical endpoints without requiring extensive retraining from scratch for each new condition.
At inference, CARDIAC-FM supports flexible input configurations: ECG-only, ECG + MRI, ECG + clinical risk scores, and ECG + MRI + clinical risk scores, enhancing its applicability in real-world clinical settings with varying data availability.
Datasets
- UK Biobank: Used for training the CARDIAC-FM model. It provided 57,609 paired ECG-cardiac MRI samples.
- Cardiovascular Health Study (CHS): An external cohort used for validation.
- Multi-Ethnic Study of Atherosclerosis (MESA): Another external cohort used for validation.
The study also evaluated predictions on a broad range of additional cardiovascular outcomes in CHS and MESA.
Key Insights and Ideas
- Multimodal Pre-training Enhances Unimodal Performance: The most striking insight is that pre-training with both ECG and MRI significantly improves the performance of the ECG-only model. This suggests that learning from MRI enriches the ECG encoder's ability to capture crucial cardiac information, even when MRI is not available at inference.
- ECG and Traditional Risk Factors are Complementary: Combining ECG features with established clinical risk scores consistently yielded additive gains in prediction accuracy, highlighting that these sources capture distinct but valuable dimensions of cardiovascular risk.
- Foundation Models Enable Efficient Adaptation: The two-stage pre-training and fine-tuning framework allows CARDIAC-FM to adapt to new clinical endpoints with minimal task-specific fine-tuning, which is crucial for real-world scenarios where labeled data for specific diseases might be scarce.
- CARDIAC-FM as a Cross-Modality Translator: The model can infer MRI-derived structural markers from ECG alone, enabling large-scale studies of cardiac structure and its relationship to outcomes in cohorts lacking imaging data.
Key Results
- Superior Performance: CARDIAC-FM consistently outperformed unimodal AI models and traditional clinical risk scores across all evaluated cohorts (UK Biobank, CHS, MESA) for predicting atrial fibrillation and heart failure.
- ECG + Risk Score Advantage: For atrial fibrillation prediction without MRI, CARDIAC-FM combining ECG with clinical risk scores achieved the highest discrimination (AUROC 0.82 in UK Biobank).
- Heart Failure Gains: Adding MRI to CARDIAC-FM provided more substantial gains for heart failure prediction compared to atrial fibrillation. The full CARDIAC-FM model (ECG + MRI + Risk Scores) achieved AUROCs of 0.82 for atrial fibrillation and 0.83 for heart failure in UK Biobank.
- Broad Applicability: CARDIAC-FM demonstrated strong generalizability across diverse populations and performed well on a wide range of cardiovascular outcomes (myocardial infarction, ischemic stroke, cardiovascular death, all-cause death) with minimal fine-tuning.
- Risk Stratification: CARDIAC-FM improved risk stratification compared to clinical risk scores alone, particularly in the CHS cohort.
Limitations and Future Work
- MRI Availability: While CARDIAC-FM can predict using ECG alone, its full potential is realized with both ECG and MRI. The authors acknowledge that MRI is expensive and not universally available, limiting its widespread clinical deployment.
- Heart Failure Phenotypes: While CARDIAC-FM showed benefits for heart failure prediction, the authors note that future improvements in imaging encoders could further strengthen performance, particularly for specific heart failure subtypes.
- Data Scarcity for Fine-tuning: The study highlights the effectiveness of minimal fine-tuning in data-scarce settings, but further research could explore optimal fine-tuning strategies for various clinical scenarios.
- Future Improvements: The authors suggest that future work could focus on enhancing imaging encoders to further improve heart failure prediction. They also emphasize the promise of multimodal foundation modeling for integrating heterogeneous clinical data into predictive cardiovascular medicine.
Summary updated