Fusing Echocardiography Images and Medical Records for Continuous Patient Stratification

Nicolas Thome, Nathan Painchaud, Pierre-Yves Courand, Pierre-Marc Jodoin, Nicolas Duchateau, Olivier BernardIEEE Transactions on Ultrasonics Ferroelectrics and Frequency Control 2024 · arXiv

Introduction

Hypertension (HT) is a global health crisis affecting over a billion people. While clinical diagnosis typically relies on discrete blood pressure thresholds, the actual physiological damage to the heart occurs along a complex, multifaceted continuum. Current AI approaches in cardiology often struggle with two major hurdles: modality silos, where models focus exclusively on either images or Electronic Health Records (EHR), and data scarcity, where specialized clinical datasets are too small for standard deep learning architectures.

To address these challenges, researchers have developed DIDACTIC, a multimodal Transformer-based framework designed to fuse structured medical records with time-series data extracted from echocardiograms. By treating patient stratification as a continuous "pathological continuum," the model identifies subtle early-warning signs of heart disease that standard clinical metrics often miss.

Overview of the DIDACTIC framework, illustrating the flow from input data to pathological characterization.

Methodology

The DIDACTIC pipeline integrates diverse data types through a unified Transformer architecture. The process begins with feature extraction, using a 3D nnU-Net to segment ultrasound sequences and extract clinically relevant time-series descriptors, such as myocardial strain and wall thickness, over a cardiac cycle.

Central to the framework is a multimodal tokenization strategy. A specialized Tabular Tokenizer processes EHR data (e.g., age, BMI, medical history), while a Time-Series Tokenizer handles the echo-derived descriptors. This allows the Transformer encoder to process heterogeneous data in a shared latent space.

Detailed architecture of the DIDACTIC pipeline, highlighting the tokenization of image and clinical attributes followed by Transformer encoding and ordinal classification.

Instead of standard classification, the model employs an ordinal classification head based on a binomial distribution. This forces the model to learn a representation where patients are arranged linearly according to disease severity (e.g., from healthy to uncontrolled hypertension), enabling continuous stratification rather than discrete categorization.

Experimental Results and Key Insights

The study utilized the CARDINAL cohort, containing 239 hypertensive patients categorized by expert cardiologists into three severity levels: White Coat, Controlled, and Uncontrolled. Despite the relatively small dataset size, the model achieved a remarkable 96.8% AUROC, significantly outperforming larger multimodal state-of-the-art methods like MMCL and IRENE, which struggled with the limited data.

Key findings from the study include:

  • The Power of the Continuum: The continuous scale revealed that some "controlled" patients are physiologically much closer to the "uncontrolled" boundary than others, providing a more granular view of risk.
  • New Pathological Markers: The model identified post-systolic relaxation degradation as a significant early indicator of worsening hypertension. This subtle change in cardiac relaxation appears before more traditional metrics, like peak strain, begin to decline.
  • Architecture over Pre-training: For small, specialized medical datasets, the inductive bias of the model architecture proved more critical than using weights pre-trained on massive, unrelated tabular datasets.

Conclusion

By fusing clinical context with image-derived time series, the DIDACTIC framework demonstrates that multimodal integration is essential for accurate patient stratification. While the current study is retrospective and limited by its sample size, it paves the way for prospective tools that could predict future cardiovascular events by identifying the earliest signs of cardiac degradation along the pathological continuum.

Summary updated

Abstract

Deep learning enables automatic and robust extraction of cardiac function descriptors from echocardiographic sequences, such as ejection fraction (EF) or strain. These descriptors provide fine-grained information that physicians consider, in conjunction with more global variables from the clinical record, to assess patients' condition. Drawing on novel Transformer models applied to tabular data, we propose a method that considers all descriptors extracted from medical records and echocardiograms to learn the representation of a cardiovascular pathology with a difficult-to-characterize continuum, namely hypertension. Our method first projects each variable into its own representation space using modality-specific approaches. These standardized representations of multimodal data are then fed to a Transformer encoder, which learns to merge them into a comprehensive representation of the patient through the task of predicting a clinical rating. This stratification task is formulated as an ordinal classification to enforce a pathological continuum in the representation space. We observe the major trends along this continuum on a cohort of 239 hypertensive patients, providing unprecedented details in the description of hypertension's impact on various cardiac function descriptors. Our analysis shows that: 1) the XTab foundation model's architecture allows to reach high performance (96.8% AUROC) even with limited data (less than 200 training samples); 2) stratification across the population is reproducible between trainings [within 5.7% of mean absolute error (MAE)]; and 3) patterns emerge in descriptors, some of which align with established physiological knowledge about hypertension, while others could pave the way for a more comprehensive understanding of this pathology. The code is available at https://github.com/creatis-myriad/didactic.
BibTeX
@misc{thome2024fusing,
  author = {Nicolas Thome and Nathan Painchaud and Pierre-Yves Courand and Pierre-Marc Jodoin and Nicolas Duchateau and Olivier Bernard},
  title = {{Fusing Echocardiography Images and Medical Records for Continuous Patient Stratification}},
  year = {2024},
  eprint = {2401.07796},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.07796},
}
Figure