MCAT: Visual Query-Based Localization of Standard Anatomical Clips in Fetal Ultrasound Videos Using Multi-Tier Class-Aware Token Transformer

Divyanshu Mishra, Pramit Saha, He Zhao, Netzahualcoyotl Hernandez-Cruz, Olga Patey, Aris T. Papageorghiou, J. Alison Noble

Introduction

Capturing standard anatomical planes in fetal ultrasound is a cornerstone of prenatal care, yet the process remains tedious and highly subjective. Sonographers often spend significant time manually scanning to identify specific views required by clinical guidelines. While AI has made strides in static frame classification, these methods often fail to account for the temporal dynamics of a continuous ultrasound sweep.

This paper introduces MCAT (Multi-Tier Class-Aware Token Transformer), a model designed for Visual Query-Based Video Clip Localization (VQ-VCL). Unlike text-based retrieval, which lacks the granularity to describe complex fetal anatomy, MCAT uses a single exemplar image (a visual query) to automatically locate the corresponding clip within a full video sweep.

Methodology

The MCAT architecture addresses the inherent noise and high visual similarity of ultrasound data through several key innovations.

The MCAT architecture overview, highlighting the multi-tier visual backbone and spatio-temporal transformer modules.

Multi-Tier Spatio-Temporal Transformer

Standard CNN backbones often lose fine-grained details in deeper layers. MCAT extracts features from multiple "tiers" of the backbone, capturing both coarse semantic information and high-resolution visual cues. These features are fused using cross-attention, where the visual query guides the contextualization of the video frames.

Efficient Tokenization and Loss Functions

To reduce computational overhead, MCAT employs class-specific spatio-temporal tokens rather than generic embeddings, resulting in a 96% reduction in token count. Furthermore, the model incorporates a Temporal Uncertainty Aware Localization Loss ($\mathcal{L}_{URL}$). Recognizing that even experts disagree on exact clip boundaries, this loss models start and end times as Gaussian distributions rather than hard binary labels, significantly improving robustness to noisy annotations.

Datasets

The model was evaluated on three distinct datasets:

  1. CAIFE Fetal Heart Sweeps: 247 clinical videos of continuous fetal heart scans.
  2. PULSE: A fetal ultrasound anomaly scan dataset containing 230 videos covering eight different anatomical planes (e.g., head, abdomen, femur).
  3. Ego4D VQ-VCL: A new benchmark created by the authors using the open-source Ego4D dataset to facilitate reproducible research in natural video domains.

Key Results and Insights

MCAT achieves state-of-the-art performance across all benchmarks, demonstrating superior accuracy and efficiency compared to existing methods like VQLOC and TubeDETR.

Quantitative comparison of MCAT against baseline methods across Heart Sweep, PULSE, and Ego4D datasets.

Key findings include:

  • Multi-tier fusion is essential: Incorporating features from multiple layers boosted performance by nearly 6%, as it preserves the low-level details necessary to distinguish nearly identical ultrasound frames.
  • Uncertainty modeling works: Embracing label noise through Gaussian distributions improved localization accuracy (mtIoU) by 14% in ablation studies.
  • Efficiency for clinical deployment: MCAT processes clips in just 2.69 seconds with minimal GPU memory (4.62 GB), making it highly suitable for resource-constrained clinical environments.

Conclusion

MCAT represents a significant step forward in automating the localization of standard anatomical views in fetal ultrasound. By combining multi-tier feature extraction with uncertainty-aware learning, the model overcomes the challenges of visual similarity and noisy labeling. Its high efficiency offers immense potential for deployment in low- and middle-income countries, where it could streamline prenatal screenings and support overextended medical staff.

Summary updated

BibTeX
@inproceedings{mishra2025mcat,
  author = {Divyanshu Mishra and Pramit Saha and He Zhao and Netzahualcoyotl Hernandez-Cruz and Olga Patey and Aris T. Papageorghiou and J. Alison Noble},
  title = {{MCAT: Visual Query-Based Localization of Standard Anatomical Clips in Fetal Ultrasound Videos Using Multi-Tier Class-Aware Token Transformer}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i27.35047},
}
Figure