Introduction
Capturing standard anatomical planes in fetal ultrasound is a cornerstone of prenatal care, yet the process remains tedious and highly subjective. Sonographers often spend significant time manually scanning to identify specific views required by clinical guidelines. While AI has made strides in static frame classification, these methods often fail to account for the temporal dynamics of a continuous ultrasound sweep.
This paper introduces MCAT (Multi-Tier Class-Aware Token Transformer), a model designed for Visual Query-Based Video Clip Localization (VQ-VCL). Unlike text-based retrieval, which lacks the granularity to describe complex fetal anatomy, MCAT uses a single exemplar image (a visual query) to automatically locate the corresponding clip within a full video sweep.
Methodology
The MCAT architecture addresses the inherent noise and high visual similarity of ultrasound data through several key innovations.
Multi-Tier Spatio-Temporal Transformer
Standard CNN backbones often lose fine-grained details in deeper layers. MCAT extracts features from multiple "tiers" of the backbone, capturing both coarse semantic information and high-resolution visual cues. These features are fused using cross-attention, where the visual query guides the contextualization of the video frames.
Efficient Tokenization and Loss Functions
To reduce computational overhead, MCAT employs class-specific spatio-temporal tokens rather than generic embeddings, resulting in a 96% reduction in token count. Furthermore, the model incorporates a Temporal Uncertainty Aware Localization Loss ($\mathcal{L}_{URL}$). Recognizing that even experts disagree on exact clip boundaries, this loss models start and end times as Gaussian distributions rather than hard binary labels, significantly improving robustness to noisy annotations.
Datasets
The model was evaluated on three distinct datasets:
- CAIFE Fetal Heart Sweeps: 247 clinical videos of continuous fetal heart scans.
- PULSE: A fetal ultrasound anomaly scan dataset containing 230 videos covering eight different anatomical planes (e.g., head, abdomen, femur).
- Ego4D VQ-VCL: A new benchmark created by the authors using the open-source Ego4D dataset to facilitate reproducible research in natural video domains.
Key Results and Insights
MCAT achieves state-of-the-art performance across all benchmarks, demonstrating superior accuracy and efficiency compared to existing methods like VQLOC and TubeDETR.
Key findings include:
- Multi-tier fusion is essential: Incorporating features from multiple layers boosted performance by nearly 6%, as it preserves the low-level details necessary to distinguish nearly identical ultrasound frames.
- Uncertainty modeling works: Embracing label noise through Gaussian distributions improved localization accuracy (mtIoU) by 14% in ablation studies.
- Efficiency for clinical deployment: MCAT processes clips in just 2.69 seconds with minimal GPU memory (4.62 GB), making it highly suitable for resource-constrained clinical environments.
Conclusion
MCAT represents a significant step forward in automating the localization of standard anatomical views in fetal ultrasound. By combining multi-tier feature extraction with uncertainty-aware learning, the model overcomes the challenges of visual similarity and noisy labeling. Its high efficiency offers immense potential for deployment in low- and middle-income countries, where it could streamline prenatal screenings and support overextended medical staff.
Summary updated