Abstract
Our results suggest that clinician-driven cardiac POCUS produces ML model LVEF estimates that correlate well with expert interpretation and echo reported LVEF.
Introduction
Left Ventricular Ejection Fraction (LVEF) is the primary metric for managing heart failure, typically measured via comprehensive hospital echocardiograms. However, these exams are resource-intensive and often involve long wait times. Point-of-care ultrasound (POCUS) offers a portable alternative at the bedside, but interpreting these images requires specialized training, and the image quality is often inferior to lab-grade equipment. While ML models have proven highly accurate at calculating LVEF from high-quality hospital echoes, their performance on "in the wild" POCUS images—which are noisier and captured by non-sonographers—remains a critical area of investigation. This study evaluates whether ML can bridge the expertise gap for clinicians using handheld devices.
Methods
The researchers utilized a previously developed ML model based on a U-Net architecture. This model performs two primary tasks: identifying the boundaries of the left ventricle frame-by-frame (segmentation) and locating the heart's apex and mitral valve (landmark detection). Using these outputs, the model calculates heart volumes and LVEF using the "method of disks" (Simpson’s rule).
The validation involved a prospective study where seven physicians and two nurse practitioners scanned 138 heart failure patients using a handheld Clarius scanner. The ML's predictions were compared against five reference standards, including visual and segmented estimates from Level III echocardiographers and formal hospital echo reports.
Results
The study used the Intraclass Correlation Coefficient (ICC) to measure reliability, with values above 0.75 considered "good." The ML model achieved an ICC of 0.772 when compared to single-video expert visual estimates. Reliability improved to 0.843 when accounting for all patient videos compared to expert segmentation. When compared against formal hospital reports, the ICC was 0.798.
Subgroup analysis revealed that performance was highly dependent on heart rhythm. For patients in normal sinus rhythm, the ICC ranged from 0.809 to 0.860. However, for those with arrhythmias such as atrial fibrillation (AF) or atrial flutter (AFL), the ICC dropped significantly to between 0.596 and 0.823.
Discussion
Several key insights emerged regarding the factors influencing ML performance:
- Temporal Integration: Agreement improved when experts viewed all videos for a patient, suggesting humans use spatial and temporal integration to filter noise that isolated ML clips may struggle with.
- Rhythm Impact: Irregular heartbeats make it difficult for both humans and algorithms to define a representative cardiac cycle, degrading model performance.
- The Scanner Paradox: Unexpectedly, higher agreement was found with scans from nurse practitioners (novices) than physicians. This was attributed to physicians scanning "less stable" patients with higher heart rates and more arrhythmias.
- BMI Robustness: High BMI did not significantly lower the correlation, suggesting the model is relatively robust to tissue depth as long as structures are visible.
Conclusion
The primary limitation identified was image feasibility; only 27% of the 1,257 captured videos were of sufficient quality for the ML model to process. This highlights the "garbage in, garbage out" challenge of clinician-driven POCUS. Future work should focus on developing real-time feedback to guide users toward better image quality and expanding the model to include other critical metrics like stroke volume and cardiac output.
Summary updated