ETAB: A Benchmark Suite for Visual Representation Learning in Echocardiography

Ahmed Alaa, Anthony Philippakis, David SontagNeurIPS 2022 · OpenReview

TL;DR

ETAB (Echocardiographic Task Adaptation Benchmark) is a new open-source suite designed to standardize how we train and evaluate AI for heart ultrasounds. By providing 31 benchmark tasks across five public datasets, it allows researchers to measure how well "pre-trained" visual models generalize to diverse clinical needs, from identifying heart structures to diagnosing diseases.

The Challenge of Echocardiography AI

Echocardiography (ultrasound of the heart) is the most common tool for diagnosing heart disease, but it presents a difficult medium for artificial intelligence. Unlike the clear, high-resolution photos found in general computer vision datasets like ImageNet, "echoes" are noisy, grayscale videos taken from specific anatomical angles (views).

Overview of datasets, views, and the transfer learning paradigm in ETAB.

The biggest hurdle in the field is the "data desert." High-quality, expert-annotated datasets are small and scattered due to privacy laws and the immense effort required for manual labeling. To solve this, researchers often use transfer learning: training a model on a large "source" task and moving it to a smaller "target" task. However, until now, there was no unified way to measure if these models actually learned useful features or just overfit to a specific hospital's data.

The ETAB Framework

The authors introduce the Echocardiographic Task Adaptation Benchmark (ETAB). More than just a dataset, it is a comprehensive evaluation framework consisting of:

  1. 31 Standardized Tasks: These are categorized into four clinical buckets: cardiac structure identification (segmentation), cardiac function estimation (regression), view classification, and clinical prediction.
  2. Task Encoding: They developed a 5-character coding system (e.g., a0-A4-E) that specifies the task, the heart view, and the dataset, making experiments reproducible and easy to track.
List of ETAB benchmark tasks and their corresponding codes.
  1. A Unified Evaluation Protocol: They propose the ETAB score, a single metric that averages a model's performance across all tasks. This allows for a "leaderboard" style comparison of different AI architectures.
  2. The ETAB Library: A PyTorch-based software package that provides a "plug-and-play" API. It handles the messy work of loading different datasets and provides a "model zoo" of 12 popular backbones (like ResNet and Vision Transformers).

Datasets

ETAB aggregates five major publicly accessible datasets, creating a "meta-dataset" for the community:

  • EchoNet (Dynamic & LVH): Over 22,000 videos/images focused on the left ventricle.
  • CAMUS: 500 patients with highly detailed annotations of the atrium and myocardial wall.
  • Unity: 1,224 samples featuring multiple views and longitudinal strain measurements.
  • TMED: 2,341 images specifically for view classification and diagnosing Aortic Stenosis.

Key Insights and Results

  • The Power of Sample Efficiency: The researchers found that pre-training on one echo task (like segmenting a ventricle) drastically improves performance on a different task when you only have a tiny amount of data (as few as 16 examples). This is a game-changer for rare heart conditions where data is scarce.
  • CNNs Still Rule the Roost: In their evaluations, classic Convolutional Neural Networks (CNNs) like MobileNet-V2 and ResNet-50 often outperformed newer, flashier Vision Transformers (ViTs). This suggests that the inductive biases of CNNs (like spatial locality) are particularly well-suited for the structured, repetitive nature of ultrasound images. MobileNet-V2 achieved the highest overall ETAB score (0.783), followed closely by ResNet-50 (0.769).
  • Segmentation Success: On the EchoNet-to-CAMUS transfer task, models achieved DICE scores (a measure of overlap) as high as 0.909, proving that features learned on large public datasets translate well to smaller, more detailed ones.
  • Label Shift is Harder than View Shift: The benchmark revealed that moving a model from one "view" to another (e.g., 4-chamber to 2-chamber) is easier for the AI than moving from one "label" to another (e.g., from segmenting the ventricle to segmenting the atrium).

Limitations and Future Work

  • 2D vs. 3D: Currently, ETAB primarily treats videos as sequences of 2D frame embeddings. The authors acknowledge that future versions should incorporate more sophisticated 3D (spatio-temporal) modeling.
  • Dataset Expansion: While ETAB uses the best public data available, the "meta-dataset" is still small compared to general computer vision benchmarks. The authors hope the community will contribute more open data over time.
  • Clinical Integration: The benchmark focuses on "visual representation," but real-world clinical utility requires integrating these models into a doctor's workflow, which involves different challenges like latency and interpretability.

Code Availability: The ETAB library is available on GitHub at github.com/ahmedmalaa/ETAB.

Summary updated

Abstract

Echocardiography is one of the most commonly used diagnostic imaging modalities in cardiology. Application of deep learning models to echocardiograms can enable automated identification of cardiac structures, estimation of cardiac function, and prediction of clinical outcomes. However, a major hindrance to realizing the full potential of deep learning is the lack of large-scale, fully curated and annotated data sets required for supervised training. High-quality pre-trained representations that can transfer useful visual features of echocardiograms to downstream tasks can help adapt deep learning models to new setups using fewer examples. In this paper, we design a suite of benchmarks that can be used to pre-train and evaluate echocardiographic representations with respect to various clinically-relevant tasks using publicly accessible data sets. In addition, we develop a unified evaluation protocol---which we call the echocardiographic task adaptation benchmark (ETAB)---that measures how well a visual representation of echocardiograms generalizes to common downstream tasks of interest. We use our benchmarking framework to evaluate state-of-the-art vision modeling pipelines. We envision that our standardized, publicly accessible benchmarks would encourage future research and expedite progress in applying deep learning to high-impact problems in cardiovascular medicine.
BibTeX
@inproceedings{alaa2022etab,
  author = {Ahmed Alaa and Anthony Philippakis and David Sontag},
  title = {{ETAB: A Benchmark Suite for Visual Representation Learning in Echocardiography}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2022},
  url = {https://openreview.net/forum?id=b0VDQiNLPy9},
}
Figure