Cross-modal autoencoder framework learns holistic representations of cardiovascular state

Adityanarayanan Radhakrishnan, Sam Friedman, Shaan Khurshid, Kenney Ng, Puneet Batra, Steven A. Lubitz, Anthony Philippakis, Caroline UhlerNature Communications 2023 · DOI

Abstract

A fundamental challenge in diagnostics is integrating multiple modalities to develop a joint characterization of physiological state. Using the heart as a model system, we develop a cross-modal autoencoder framework for integrating distinct data modalities and constructing a holistic representation of cardiovascular state. In particular, we use our framework to construct such cross-modal representations from cardiac magnetic resonance images (MRIs), containing structural information, and electrocardiograms (ECGs), containing myoelectric information. We leverage the learned cross-modal representation to (1) improve phenotype prediction from a single, accessible phenotype such as ECGs; (2) enable imputation of hard-to-acquire cardiac MRIs from easy-to-acquire ECGs; and (3) develop a framework for performing genome-wide association studies in an unsupervised manner. Our results systematically integrate distinct diagnostic modalities into a common representation that better characterizes physiologic state.

Introduction

In cardiology, clinicians typically rely on isolated data streams: electrocardiograms (ECGs) for rapid electrical assessment and cardiac magnetic resonance imaging (MRI) for detailed physical structure. Integrating these modalities—the "rhythm" and the "geometry" of the heart—remains a fundamental challenge. Furthermore, traditional genome-wide association studies (GWAS) are often limited by manually defined phenotypes, which may overlook complex, subtle patterns. This study introduces a cross-modal autoencoder framework designed to learn a unified, "holistic" representation of cardiovascular health from multi-modal data.

Methods

Architecture and Training Pipeline

The core of the framework is a cross-modal autoencoder that maps disparate inputs into a shared 256-dimensional latent space. The architecture employs modality-specific encoders: a 1D convolutional encoder for ECG signals and a 2D+time encoder for cardiac MRI videos.

Cross-modal training pipeline illustrating the integration of ECG and MRI data into a shared latent space via contrastive and reconstruction losses.

Training utilizes a dual-loss approach. A reconstruction loss ensures that modality-specific decoders can recreate the original data from the latent representation. Simultaneously, a contrastive loss forces ECG and MRI embeddings from the same individual to be proximal in the latent space while separating those from different individuals. To ensure the model remains functional when only one modality is available at test time, "coordinate dropout" was implemented during training.

Dataset and Scale

The model was trained using the UK Biobank (UKB) dataset, consisting of 38,686 paired samples of median 12-lead ECGs and 50-frame cine MRIs. The data was partitioned into training (27,160), validation (6,780), and a held-out test set (4,746).

Results

Holistic Latent Representations

t-SNE visualizations confirmed that the cross-modal approach successfully "mixed" ECG and MRI data into a modality-independent representation. A key finding was the impact of unlabeled data: increasing the number of unlabeled ECG-MRI pairs for pre-training improved downstream performance twice as effectively as increasing labeled samples for specific tasks.

Unsupervised Genetic Discovery

The framework enables "unsupervised GWAS" by performing analysis directly on the latent space vectors. Using MANOVA and Iterative Nullspace Projection (INLP) to remove confounders like age and sex, the model identified significant genetic markers without pre-defined labels. For instance, it successfully identified the NOS1AP variant associated with the QT interval, effectively discovering genetic drivers of heart shape and rhythm autonomously.

GWAS and clustering of the cross-modal latent space, showing how the model identifies genetic variants (SNPs) associated with cardiovascular state.

Phenotype Prediction and Cross-Modal Translation

The cross-modal latent space outperformed supervised deep learning models in predicting clinical traits such as age, sex, and BMI. Additionally, the model demonstrated the ability to generate synthetic MRIs from ECG inputs. These reconstructions captured physical traits like Left Ventricular Mass ($R^2 = 0.458$) and Right Ventricular End-Diastolic Volume ($R^2 = 0.265$), providing a pathway to infer complex structural information from accessible electrical signals.

Discussion

This framework demonstrates the power of integrating diagnostic modalities into a common representation. However, limitations remain, particularly regarding the diversity of the UK Biobank population, which lacks significant racial and socioeconomic representation. Future research must validate these models in more diverse cohorts and explore causally-grounded methods to further disentangle genetic and environmental signals. Clinical deployment will also require testing against the noise and variability inherent in real-world hospital data.

Summary updated

Abstract

A fundamental challenge in diagnostics is integrating multiple modalities to develop a joint characterization of physiological state. Using the heart as a model system, we develop a cross-modal autoencoder framework for integrating distinct data modalities and constructing a holistic representation of cardiovascular state. In particular, we use our framework to construct such cross-modal representations from cardiac magnetic resonance images (MRIs), containing structural information, and electrocardiograms (ECGs), containing myoelectric information. We leverage the learned cross-modal representation to (1) improve phenotype prediction from a single, accessible phenotype such as ECGs; (2) enable imputation of hard-to-acquire cardiac MRIs from easy-to-acquire ECGs; and (3) develop a framework for performing genome-wide association studies in an unsupervised manner. Our results systematically integrate distinct diagnostic modalities into a common representation that better characterizes physiologic state.
BibTeX
@inproceedings{radhakrishnan2023cross,
  author = {Adityanarayanan Radhakrishnan and Sam Friedman and Shaan Khurshid and Kenney Ng and Puneet Batra and Steven A. Lubitz and Anthony Philippakis and Caroline Uhler},
  title = {{Cross-modal autoencoder framework learns holistic representations of cardiovascular state}},
  booktitle = {Nature Communications},
  year = {2023},
  doi = {10.1038/s41467-023-38125-0},
}
Figure