PanEcho: Complete AI-enabled echocardiography interpretation with multi-task deep learning

Gregory Holste, Evangelos K. Oikonomou, Márton Tokodi, Attila Kovács, Zhangyang Wang, Rohan KheramedRxiv 2024 · DOI

TL;DR

PanEcho is a "view-agnostic" multi-task deep learning model capable of simultaneously performing 39 different echocardiographic interpretation tasks. By training on over a million videos, the system achieves expert-level accuracy across diverse clinical findings and serves as a robust "foundation model" that can be easily adapted to new datasets or abbreviated point-of-care protocols.

What Problem Does This Paper Address?

Echocardiography (cardiac ultrasound) is the most common way to visualize heart structure and function, but it is bottlenecked by human factors. Interpreting a study requires a highly skilled cardiologist to integrate information from dozens of different "views" (angles) and imaging modes (like B-mode or color Doppler).

While AI has shown promise in this field, previous models have been "siloed"—designed for a single specific task (like measuring Ejection Fraction) using a single specific view (like the Apical 4-chamber). In the real world, clinical protocols vary, and a model that breaks when given the "wrong" view is difficult to deploy. PanEcho addresses this by creating a holistic, all-in-one system that handles the entire breadth of a standard echo report regardless of which views are available.

Methods

The authors utilized a "2+1D" decoupled architecture designed for both spatial and temporal flexibility:

  • Image Encoder: A ConvNeXt-T (convolutional neural network) processes individual video frames to create spatial embeddings.
  • Temporal Transformer: These embeddings are fed into a Transformer encoder with self-attention layers to model the motion and timing patterns across 16-frame video clips.
  • Multi-Task Heads: The resulting video-level representation is passed to 39 separate output "heads"—18 for classification (e.g., "Is there aortic stenosis?") and 21 for regression (e.g., "What is the left ventricular diameter?").

Crucially, the model is view-agnostic. During training, it learns to extract relevant features from any standard echo view. At test time, predictions from all videos in a patient's study are averaged to produce a final, comprehensive report.

Datasets

The scale of the data used is a major highlight of this work:

  • Development/Internal Validation: 1.23 million echocardiographic videos from ~34,000 studies (26,067 patients) at the Yale-New Haven Health System (YNHHS).
  • External Validation (Generalizability):
    • EchoNet-LVH: 12,000 videos from Stanford focused on heart structure.
  • EchoNet-Dynamic: Over 10,000 videos from Stanford focused on heart function.
  • Transfer Learning: EchoNet-Pediatric, a dataset of ~7,000 pediatric echoes used to test how well the model adapts to "out-of-domain" populations.

Key Insights and Ideas

  • Emergent Clinical Logic: One of the most striking findings was the model's internal "view relevance." Without being explicitly told which view is best for which measurement, PanEcho's internal scores aligned perfectly with international clinical guidelines (e.g., it "learned" that the Parasternal Long Axis view is the gold standard for measuring the aortic root).
  • Foundation Model Properties: The researchers demonstrated that PanEcho isn't just a classifier; it's a powerful feature extractor. When fine-tuning for new tasks (like pediatric heart measurements), a PanEcho-pretrained model converged faster and achieved higher accuracy than models starting from ImageNet or general video datasets (Kinetics-400).
  • Resilience to "Limited" Studies: The model maintained high performance even when restricted to an abbreviated 5-video protocol. This suggests it could be highly effective for rapid screening in emergency rooms or using handheld devices where a full 50-video study isn't possible.

Key Results

  • High Diagnostic Accuracy: Across 18 classification tasks, the model achieved a median AUC of 0.91. It was particularly strong in detecting severe aortic stenosis (AUC 0.99) and systolic dysfunction (AUC 0.98).
  • Precise Measurements: For the 21 regression tasks, it achieved a median normalized Mean Absolute Error (MAE) of 0.13. Its estimate of Left Ventricular Ejection Fraction (LVEF) had an MAE of only 4.4% internally and 5.5% on external Stanford data.
  • Superior Transfer Learning: On the pediatric dataset, PanEcho reached peak performance in just ~5 epochs, significantly outperforming standard computer vision initializations and even domain-specific models like EchoCLIP.

Limitations and Future Work

  • Simple Aggregation: Currently, the model uses a simple average to combine predictions from different videos in a study. The authors note that "learned" attention—where the model adaptively weights the most relevant videos—could further improve accuracy.
  • Lack of Segmentation: Unlike some AI tools that draw "masks" over the heart chambers, PanEcho is a direct-to-prediction model. While this makes it more robust to image noise, it provides less visual "proof" of its measurements to the clinician.
  • Prospective Testing: The study is retrospective. The next step is "shadow testing" in real-world clinical workflows to see how it performs as a live "pre-reader" for cardiologists.

Summary updated

Abstract

We report an AI system that automatically interprets echocardiograms, maintaining high accuracy across geography and time from complete and limited studies. PanEcho may be used as an adjunct reader in echocardiography labs or rapid AI-enabled screening tool in point-of-care settings.
BibTeX
@misc{holste2024panecho,
  author = {Gregory Holste and Evangelos K. Oikonomou and Márton Tokodi and Attila Kovács and Zhangyang Wang and Rohan Khera},
  title = {{PanEcho: Complete AI-enabled echocardiography interpretation with multi-task deep learning}},
  year = {2024},
  doi = {10.1101/2024.11.16.24317431},
  howpublished = {medRxiv},
}
Figure