Self-supervised contrastive learning of echocardiogram videos enables label-efficient cardiac disease diagnosis

Gregory Holste, Evangelos K. Oikonomou, Bobak J. Mortazavi, Zhangyang Wang, Rohan Khera

TL;DR

EchoCLR is a new self-supervised learning (SSL) framework designed specifically for echocardiogram videos, overcoming the "label bottleneck" in medical AI. By leveraging multi-instance contrastive learning and a temporal frame-ordering task, it achieves high diagnostic accuracy for cardiac diseases using as little as 1% of the labeled data required by traditional methods.

What Problem Does This Paper Address?

Deep learning has shown immense promise in medical imaging, but it has a massive "data hunger" problem. Training a model to recognize complex conditions like Aortic Stenosis (AS) or Left Ventricular Hypertrophy (LVH) typically requires tens of thousands of expert-labeled examples. In medicine, these labels are expensive, time-consuming, and require specialized expertise to produce.

While self-supervised learning (SSL)—where models learn from unlabeled data by solving "pretext" puzzles—has worked well for natural images (like photos of cats and dogs), it struggles with echocardiograms. Ultrasound videos are noisy, low-resolution, and highly dependent on the sonographer's skill. Standard SSL techniques often rely on heavy data augmentations (like cropping or color distortion) that can destroy the subtle clinical signals in an ultrasound. Furthermore, most medical SSL research focuses on static 2D images, ignoring the critical temporal information (the heart's motion) contained in a video.

Methods

The authors developed EchoCLR, a spatiotemporal SSL approach tailored for ultrasound videos. It introduces two key innovations to the standard contrastive learning (SimCLR) framework:

  1. Multi-Instance Contrastive Learning: Instead of taking one video and artificially distorting it to create a "positive pair," EchoCLR uses two different videos of the same patient's heart taken during the same exam. This forces the model to learn the underlying anatomical and functional features of that specific patient, rather than just memorizing pixel-level patterns.
  2. Frame Re-ordering Pretext Task: To capture the heart's motion, the authors added a second task. The model is given a set of shuffled video frames and must predict their original chronological order. This "temporal coherence" task ensures the model understands the phases of the cardiac cycle.

The backbone architecture used is a 3D-ResNet18, which is capable of processing spatial and temporal dimensions simultaneously.

Datasets

The study utilized a large-scale dataset from Yale New Haven Hospital:

  • Development & Internal Testing: 10,000 TTE (transthoracic echocardiogram) studies from 2016–2020.
  • External Testing: 2,500 studies from 2021, serving as a temporally distinct test set to ensure the model generalizes to new data.
  • Focus View: The researchers focused on the Parasternal Long Axis (PLAX) view, the most common viewpoint in echocardiography.
  • Labels: Expert cardiologist interpretations for LVH and AS were used as the ground truth for fine-tuning and evaluation.

Key Insights and Ideas

  • The "Low-Data" Advantage: The most striking insight is that EchoCLR's benefits are greatest when labeled data is scarcest. It acts as a "force multiplier" for small datasets.
  • Augmentation vs. Reality: In noisy modalities like ultrasound, real-world variations (different video captures of the same heart) are superior to synthetic augmentations for contrastive learning.
  • The Saturation Point: The authors identified a "saturation point" at approximately 25% of the training data (~1,000 studies). Beyond this point, standard transfer learning (using weights from natural video datasets like Kinetics-400) performs similarly to EchoCLR. This suggests that SSL is primarily a tool for rapid adaptation in data-starved environments.
  • Improved Interpretability: Using Grad-CAM saliency maps, the authors found that EchoCLR-pretrained models naturally "look" at more clinically relevant areas (like the aortic valve) compared to models trained with standard transfer learning, which had more diffuse and less relevant attention patterns.

Key Results

EchoCLR significantly outperformed standard transfer learning and basic SSL in the "low-data regime":

  • Severe Aortic Stenosis (AS): Using only 1% of labeled data (just 53 studies), EchoCLR achieved an AUROC of 0.82, compared to just 0.61 for standard transfer learning.
  • Left Ventricular Hypertrophy (LVH): With 10% of labeled data (519 studies), EchoCLR reached an AUROC of 0.72, significantly beating the 0.61 achieved by transfer learning.
  • Ablation Success: The authors proved that both the multi-instance sampling and the frame-ordering task were necessary; removing either component led to a measurable drop in diagnostic performance.

Limitations and Future Work

  • Single-View Limitation: This study was restricted to the PLAX view. Real-world diagnosis often involves integrating multiple views (apical, short-axis, etc.). The authors suggest that future SSL methods should learn joint representations across multiple viewpoints.
  • Saturation at Scale: Because EchoCLR converges with other methods once labels are plentiful, its primary utility is for rare diseases or clinical settings where large-scale labeling is impossible.
  • Multimodal Potential: The authors propose that future work could link echocardiogram videos with other data types, such as ECGs or electronic health records, to create a more holistic "foundation model" for cardiology.

Summary updated

Abstract

Advances in self-supervised learning (SSL) have shown that self-supervised pretraining on medical imaging data can provide a strong initialization for downstream supervised classification and segmentation. Given the difficulty of obtaining expert labels for medical image recognition tasks, such an "in-domain" SSL initialization is often desirable due to its improved label efficiency over standard transfer learning. However, most efforts toward SSL of medical imaging data are not adapted to video-based medical imaging modalities. With this progress in mind, we developed a self-supervised contrastive learning approach, EchoCLR, catered to echocardiogram videos with the goal of learning strong representations for efficient fine-tuning on downstream cardiac disease diagnosis. EchoCLR leverages (i) distinct videos of the same patient as positive pairs for contrastive learning and (ii) a frame re-ordering pretext task to enforce temporal coherence. When fine-tuned on small portions of labeled data (as few as 51 exams), EchoCLR pretraining significantly improved classification performance for left ventricular hypertrophy (LVH) and aortic stenosis (AS) over other transfer learning and SSL approaches across internal and external test sets. For example, when fine-tuning on 10% of available training data (519 studies), an EchoCLR-pretrained model achieved 0.72 AUROC (95% CI: [0.69, 0.75]) on LVH classification, compared to 0.61 AUROC (95% CI: [0.57, 0.64]) with a standard transfer learning approach. Similarly, using 1% of available training data (53 studies), EchoCLR pretraining achieved 0.82 AUROC (95% CI: [0.79, 0.84]) on severe AS classification, compared to 0.61 AUROC (95% CI: [0.58, 0.65]) with transfer learning. EchoCLR is unique in its ability to learn representations of medical videos and demonstrates that SSL can enable label-efficient disease classification from small, labeled datasets.
BibTeX
@inproceedings{holste2024self,
  author = {Gregory Holste and Evangelos K. Oikonomou and Bobak J. Mortazavi and Zhangyang Wang and Rohan Khera},
  title = {{Self-supervised contrastive learning of echocardiogram videos enables label-efficient cardiac disease diagnosis}},
  booktitle = {Communications},
  year = {2024},
  doi = {10.1038/s43856-024-00538-3},
}
Figure