Vision-language models for decoding provider attention during neonatal resuscitation

Felipe Parodi, Jordan K. Matelsky, Alejandra Regla-Vargas, Elizabeth E. Foglia, Charis Lim, Danielle Weinberg, Konrad P. Kording, Heidi M. Herrick, Michael L. PlattWorkshops @ CVPR 2024 · arXiv

Abstract

Neonatal resuscitations demand an exceptional level of attentiveness from providers, who must process multiple streams of information simultaneously. Gaze strongly influences decision making; thus, understanding where a provider is looking during neonatal resuscitations could inform provider training, enhance real-time decision support, and improve the design of delivery rooms and neonatal intensive care units (NICUs). Current approaches to quantifying neonatal providers' gaze rely on manual coding or simulations, which limit scalability and utility. Here, we introduce an automated, real-time, deep learning approach capable of decoding provider gaze into semantic classes directly from first-person point-of-view videos recorded during live resuscitations. Combining state-of-the-art, real-time segmentation with vision-language models (CLIP), our low-shot pipeline attains 91% classification accuracy in identifying gaze targets without training. Upon fine-tuning, the performance of our gaze-guided vision transformer exceeds 98% accuracy in gaze classification, approaching human-level precision. This system, capable of real-time inference, enables objective quantification of provider attention dynamics during live neonatal resuscitation. Our approach offers a scalable solution that seamlessly integrates with existing infrastructure for data-scarce gaze analysis, thereby offering new opportunities for understanding and refining clinical decision making.

Introduction

Neonatal resuscitation is a high-stakes, "controlled chaos" environment where lead providers must simultaneously monitor vital signs, perform complex procedures, and coordinate a multidisciplinary team. In these critical moments, where a provider directs their attention—their "gaze"—serves as a vital proxy for their underlying decision-making process.

Historically, analyzing gaze in clinical settings required researchers to manually code hours of egocentric (first-person) video frame by frame, a process that is inherently unscalable and precludes real-time application. While eye-tracking hardware provides raw spatial coordinates, it lacks the semantic context to identify the specific objects of interest (e.g., distinguishing between the infant and a nearby monitor). This research addresses the challenge of semantic gaze analysis: the automated, real-time labeling of attention targets during live, high-pressure medical interventions.

Methodology

The proposed pipeline is designed for high-speed inference and accuracy, even in data-scarce clinical environments. The process follows four primary stages:

  1. Data Capture: Providers wear Tobii Pro eye-tracking glasses to record 1080p egocentric video and capture real-time gaze coordinates.
  2. Subject Isolation: To focus the model on relevant features, the system utilizes MobileSAM (a lightweight Segment Anything Model) to generate a mask of the object at the gaze coordinate. Simultaneously, a 128x128 pixel crop centered on the gaze is created.
  3. Classification: These "Crop + Mask" pairs are processed by a classifier. The study evaluated several architectures, including Zero-shot/Few-shot CLIP (using natural language understanding) and fine-tuned models like MobileViT and ResNet50.
  4. Interpretability: Grad-CAM was employed to generate attention heatmaps, ensuring the models focused on clinical targets rather than background artifacts.
Overview of the automated gaze analysis pipeline, from video input to semantic classification.

The EDIR Dataset

The study introduced the Egocentric Dataset of Infant Resuscitation (EDIR), comprising approximately 14,700 unique frames recorded during nine live neonatal resuscitation sessions at the Children’s Hospital of Philadelphia. Unlike datasets based on simulations or manikins, EDIR captures the complexity and noise of real-world clinical environments. Frames are labeled across seven semantic categories: Infant, Vitals Monitor, Video Laryngoscope Screen, Airway Equipment, Airway Provider, Non-Team Member, and Other Physical Objects.

Results and Discussion

The experimental results demonstrate that the synergy of localized crops and segmentation masks significantly enhances performance by providing spatial context while filtering out background distractions.

Performance Metrics

  • High Accuracy: The fine-tuned MobileViT model achieved a 93.02% Top-1 accuracy and 98.74% Top-3 accuracy, approaching human-level precision.
  • Robust Low-Shot Learning: The Tip-Adapted CLIP model, trained on only 16 images, reached 91.67% Top-3 accuracy, highlighting the utility of foundation models in niche clinical tasks.
  • Real-Time Efficiency: The pipeline achieves inference speeds of approximately 138 FPS on high-end GPUs, comfortably exceeding the 25 FPS capture rate of the eye-tracking hardware.

Interpretability

Visualization through Grad-CAM confirmed that the models, particularly MobileViT and CLIP-based architectures, successfully localized their attention on the intended clinical objects. This interpretability is crucial for establishing trust in clinical AI systems.

Comparison of attention heatmaps (Grad-CAM) across different model architectures for a CMAC-Screen gaze target.

Conclusion and Future Work

This automated pipeline transforms semantic gaze analysis from a labor-intensive manual task into a scalable, real-time tool for clinical training and decision support. However, challenges remain regarding hardware dependency on initial calibration and the ethical implications of recording live resuscitations. Future work will focus on validating the system across multiple institutions and moving toward more inherently explainable AI frameworks to facilitate clinical certification.

Summary updated

Abstract

Neonatal resuscitations demand an exceptional level of attentiveness from providers, who must process multiple streams of information simultaneously. Gaze strongly influences decision making; thus, understanding where a provider is looking during neonatal resuscitations could inform provider training, enhance real-time decision support, and improve the design of delivery rooms and neonatal intensive care units (NICUs). Current approaches to quantifying neonatal providers' gaze rely on manual coding or simulations, which limit scalability and utility. Here, we introduce an automated, real-time, deep learning approach capable of decoding provider gaze into semantic classes directly from first-person point-of-view videos recorded during live resuscitations. Combining state-of-the-art, real-time segmentation with vision-language models (CLIP), our low-shot pipeline attains 91\% classification accuracy in identifying gaze targets without training. Upon fine-tuning, the performance of our gaze-guided vision transformer exceeds 98\% accuracy in gaze classification, approaching human-level precision. This system, capable of real-time inference, enables objective quantification of provider attention dynamics during live neonatal resuscitation. Our approach offers a scalable solution that seamlessly integrates with existing infrastructure for data-scarce gaze analysis, thereby offering new opportunities for understanding and refining clinical decision making.
BibTeX
@inproceedings{parodi2024vision,
  author = {Felipe Parodi and Jordan K. Matelsky and Alejandra Regla-Vargas and Elizabeth E. Foglia and Charis Lim and Danielle Weinberg and Konrad P. Kording and Heidi M. Herrick and Michael L. Platt},
  title = {{Vision-language models for decoding provider attention during neonatal resuscitation}},
  booktitle = {Workshops @ CVPR},
  year = {2024},
}
Figure