Abstract
A fundamental challenge in diagnostics is integrating multiple modalities to develop a joint characterization of physiological state. Using the heart as a model system, we develop a cross-modal autoencoder framework for integrating distinct data modalities and constructing a holistic representation of cardiovascular state. In particular, we use our framework to construct such cross-modal representations from cardiac magnetic resonance images (MRIs), containing structural information, and electrocardiograms (ECGs), containing myoelectric information. We leverage the learned cross-modal representation to (1) improve phenotype prediction from a single, accessible phenotype such as ECGs; (2) enable imputation of hard-to-acquire cardiac MRIs from easy-to-acquire ECGs; and (3) develop a framework for performing genome-wide association studies in an unsupervised manner. Our results systematically integrate distinct diagnostic modalities into a common representation that better characterizes physiologic state.
Introduction
In cardiology, clinicians typically rely on isolated data streams: electrocardiograms (ECGs) for rapid electrical assessment and cardiac magnetic resonance imaging (MRI) for detailed physical structure. Integrating these modalities—the "rhythm" and the "geometry" of the heart—remains a fundamental challenge. Furthermore, traditional genome-wide association studies (GWAS) are often limited by manually defined phenotypes, which may overlook complex, subtle patterns. This study introduces a cross-modal autoencoder framework designed to learn a unified, "holistic" representation of cardiovascular health from multi-modal data.
Methods
Architecture and Training Pipeline
The core of the framework is a cross-modal autoencoder that maps disparate inputs into a shared 256-dimensional latent space. The architecture employs modality-specific encoders: a 1D convolutional encoder for ECG signals and a 2D+time encoder for cardiac MRI videos.
Training utilizes a dual-loss approach. A reconstruction loss ensures that modality-specific decoders can recreate the original data from the latent representation. Simultaneously, a contrastive loss forces ECG and MRI embeddings from the same individual to be proximal in the latent space while separating those from different individuals. To ensure the model remains functional when only one modality is available at test time, "coordinate dropout" was implemented during training.
Dataset and Scale
The model was trained using the UK Biobank (UKB) dataset, consisting of 38,686 paired samples of median 12-lead ECGs and 50-frame cine MRIs. The data was partitioned into training (27,160), validation (6,780), and a held-out test set (4,746).
Results
Holistic Latent Representations
t-SNE visualizations confirmed that the cross-modal approach successfully "mixed" ECG and MRI data into a modality-independent representation. A key finding was the impact of unlabeled data: increasing the number of unlabeled ECG-MRI pairs for pre-training improved downstream performance twice as effectively as increasing labeled samples for specific tasks.
Unsupervised Genetic Discovery
The framework enables "unsupervised GWAS" by performing analysis directly on the latent space vectors. Using MANOVA and Iterative Nullspace Projection (INLP) to remove confounders like age and sex, the model identified significant genetic markers without pre-defined labels. For instance, it successfully identified the NOS1AP variant associated with the QT interval, effectively discovering genetic drivers of heart shape and rhythm autonomously.
Phenotype Prediction and Cross-Modal Translation
The cross-modal latent space outperformed supervised deep learning models in predicting clinical traits such as age, sex, and BMI. Additionally, the model demonstrated the ability to generate synthetic MRIs from ECG inputs. These reconstructions captured physical traits like Left Ventricular Mass ($R^2 = 0.458$) and Right Ventricular End-Diastolic Volume ($R^2 = 0.265$), providing a pathway to infer complex structural information from accessible electrical signals.
Discussion
This framework demonstrates the power of integrating diagnostic modalities into a common representation. However, limitations remain, particularly regarding the diversity of the UK Biobank population, which lacks significant racial and socioeconomic representation. Future research must validate these models in more diverse cohorts and explore causally-grounded methods to further disentangle genetic and environmental signals. Clinical deployment will also require testing against the noise and variability inherent in real-world hospital data.
Summary updated