Abstract
Cardiac ultrasound imaging is used to diagnose various heart diseases. Common analysis pipelines involve manual processing of the video frames by expert clinicians. This suffers from intra- and inter-observer variability. We propose a novel approach to ultrasound video analysis using a transformer architecture based on a Residual Auto-Encoder Network and a BERT model adapted for token classification. This enables videos of any length to be processed. We apply our model to the task of End-Systolic (ES) and End-Diastolic (ED) frame detection and the automated computation of the left ventricular ejection fraction. We achieve an average frame distance of 3.36 frames for the ES and 7.17 frames for the ED on videos of arbitrary length. Our end-to-end learnable approach can estimate the ejection fraction with a MAE of 5.95 and $R^2$ of 0.52 in 0.15 s per video, showing that segmentation is not the only way to predict ejection fraction.
Introduction
Measuring the Left Ventricular Ejection Fraction (LVEF)—the percentage of blood pumped out of the heart with each beat—is a cornerstone of cardiac diagnosis. However, the current clinical "gold standard" is labor-intensive: clinicians must manually identify the heart's most expanded (End-Diastolic, ED) and most contracted (End-Systolic, ES) frames, trace the heart's boundaries, and calculate the volume.
This process is slow, highly subjective (leading to high inter-observer variability), and usually only analyzes a single heartbeat. Existing AI solutions often rely on complex multi-step pipelines or are limited to processing fixed-length video clips, which fails to capture the natural variation in heart rates and video durations found in real clinical settings. The Ultrasound Video Transformer (UVT) addresses these challenges by treating cardiac ultrasound videos as sequences of tokens, much like natural language.
Methodology
The UVT architecture treats video frames as "tokens" in a sequence, allowing for global spatio-temporal reasoning across videos of any length. The system consists of three main components:
- Dimensionality Reduction: Using a ResNet Auto-Encoder (ResNetAE), the model compresses high-resolution ultrasound frames into 1024-dimensional feature vectors.
- Spatio-Temporal Reasoning: A Bidirectional Encoder Representations from Transformers (BERT) model, consisting of 16 encoders, processes the sequence of frame embeddings. This allows the model to "look" across the entire video to understand the relationship between different heartbeats.
- Regression Heads: Two specialized branches sit at the end of the BERT model. One predicts the indices of the ES and ED frames (Systole-Diastole Regressor), while the other directly regresses the LVEF value (Ejection Fraction Regressor).
To train the model effectively on limited data, the authors used data augmentation techniques like Mirroring (looping the heart cycle) and Guided Random Sampling to ensure the model learned to identify cardiac phases regardless of where they appeared in the clip.
Results and Analysis
The model was evaluated using the EchoNet-Dynamic dataset, a large-scale public benchmark containing 10,030 echocardiogram videos. The UVT achieved a Mean Absolute Error (MAE) of 5.95 for LVEF estimation, performing within the range of expert clinician inter-observer variability (typically 7.6% to 13.9%).
Key findings from the study include:
- Segmentation is Optional: The transformer learns to estimate LVEF directly from temporal features and compressed image data, bypassing the need for computationally expensive pixel-level masks.
- Temporal Context: By applying BERT, the researchers successfully captured long-range temporal dependencies. Unlike LSTMs, the transformer maintains a global view of the entire video, which is crucial for identifying multiple heartbeats.
- Real-Time Efficiency: The entire inference process takes just 0.15 seconds per video. This is significantly faster than previous state-of-the-art methods and enables potential real-time use during live ultrasound exams.
Conclusion
The Ultrasound Video Transformer demonstrates that end-to-end temporal modeling can match the accuracy of traditional segmentation-heavy pipelines while offering superior speed and flexibility. Future work will focus on clinical validation against Cardiac MRI, expanding the model to handle multi-view analysis for biplane volume estimation, and investigating minor prediction biases in exceptionally long video sequences.
Summary updated