Subtle Signals: Video-based Detection of Infant Non-nutritive Sucking as a Neurodevelopmental Cue

Shaotong Zhu, Michael Wan, Sai Kumar Reddy Manne, Emily Zimmerman, Sarah Ostadabbas

Introduction

Non-nutritive sucking (NNS)—the rhythmic sucking behavior infants exhibit on pacifiers or fingers—serves as a critical "vital sign" for early neurodevelopment. In the Neonatal Intensive Care Unit (NICU), NNS is a primary indicator of feeding readiness in preterm infants. In older infants, it provides insights into motor development and has been associated with a reduced risk of Sudden Infant Death Syndrome (SIDS).

Despite its clinical value, current assessment methods are flawed. Manual assessment is subjective and labor-intensive, while pressure-sensing pacifiers are expensive and can interfere with natural behavior. This research presents a contactless, vision-based system designed to automate NNS monitoring using standard baby monitor footage, enabling objective assessment in both clinical and home environments.

Methodology

The proposed system utilizes a two-stage pipeline for action recognition and temporal segmentation.

  1. Preprocessing and Motion Amplification: To isolate the infant's face, the system employs RetinaFace for detection and a MOSSE tracker for stabilization. Because NNS involves extremely subtle jaw and mouth movements, raw RGB data often fails to provide sufficient signal. The researchers address this by calculating dense optical flow, converted into the HSV color space, to highlight rhythmic motion vectors.
  2. Spatiotemporal Modeling: The core architecture is a 2D-1D CNN-LSTM. A ResNet18 backbone extracts spatial features from optical flow frames, which are then processed by a Long Short-Term Memory (LSTM) network to identify the temporal rhythm of sucking.
  3. Action Segmentation: To process long-form, untrimmed video, a Multi-Stage Temporal Convolutional Network (MS-TCN) is used to "stitch" classified segments into discrete events, identifying the precise start and end of sucking bursts.

Data and Experimental Setup

The study introduced two novel datasets to address the lack of infant-specific computer vision data. The NNS Clinical In-Crib Dataset comprises 183 hours of infrared footage from 19 infants, capturing real-world challenges such as low light, caregiver interference, and infant movement. This was supplemented by an "In-the-Wild" dataset of YouTube videos to test generalizability across different camera angles and lighting conditions.

Figure 1: Examples of infant monitoring footage from the clinical and "in-the-wild" datasets, illustrating the variety of lighting, poses, and occlusions the model must navigate.

Results and Discussion

The vision-based approach demonstrated high efficacy across multiple metrics. The ResNet18-LSTM model achieved 95.8% classification accuracy on the clinical dataset. Notably, models trained on raw RGB data performed near random (approx. 50%), confirming that motion amplification via optical flow is essential for detecting these subtle cues.

In terms of temporal segmentation, the MS-TCN model achieved an average precision of 93.5%. The system successfully identified the characteristic "burst-pause" patterns of NNS, where infants typically perform 6–12 sucks at a frequency of approximately 2 Hz.

Figure 2: Temporal segmentation results showing the alignment between detected NNS events and the presence of a pacifier over time.

Testing on the "In-the-Wild" dataset yielded 92.3% accuracy, suggesting that the model is robust and not overfitted to specific laboratory or hospital settings. While performance slightly decreased in highly challenging conditions (e.g., heavy occlusion or swinging cribs), the results represent a significant step toward automated neurodevelopmental screening.

Conclusion

By combining dense optical flow with spatiotemporal deep learning, this research provides a viable, contactless alternative to intrusive NNS monitoring. Future work will focus on quantifying specific metrics such as sucking frequency and amplitude, potentially offering a fully automated tool for early detection of developmental delays via standard home monitoring systems.

Summary updated

BibTeX
@article{zhu2024subtle,
  author = {Shaotong Zhu and Michael Wan and Sai Kumar Reddy Manne and Emily Zimmerman and Sarah Ostadabbas},
  title = {{Subtle Signals: Video-based Detection of Infant Non-nutritive Sucking as a Neurodevelopmental Cue}},
  journal = {Comput. Vis. Image Underst.},
  year = {2024},
  doi = {10.1016/j.cviu.2024.104081},
}
Figure