Localised photoplethysmography imaging for heart rate estimation of pre-term infants in the clinic

Gabrielle Green, Sitthichok Chaichulee, Mauricio Villarroel, João Jorge, Carlos Arteta, Andrew Zisserman, Lionel Tarassenko, Kenny McCormick2018 · DOI

TL;DR

This paper presents a novel non-contact system for estimating the heart rate of preterm infants using video. By extending a convolutional neural network (CNN) to detect specific body parts (head, torso, diaper), they can extract more robust photoplethysmographic imaging (PPGi) signals, leading to a significant reduction in error and increased uptime compared to previous methods.

What Problem Does This Paper Address?

Monitoring vital signs, particularly heart rate, is crucial for the well-being of preterm infants in Neonatal Intensive Care Units (NICUs). Traditional methods, like pulse oximetry and ECG, rely on adhesive sensors that can irritate fragile infant skin, increasing the risk of infection and discomfort. Furthermore, these sensors can restrict movement and require constant repositioning. The challenge is to develop a non-contact method that can reliably and continuously monitor heart rate in this sensitive population without causing harm or inconvenience. Previous non-contact approaches have shown promise but often struggle with signal quality and continuous monitoring, especially in a clinical setting with varying lighting and infant movement.

Methods

The core of this work builds upon a previously developed CNN for patient detection and skin segmentation. The key innovation here is the extension of this CNN to perform body part detection, specifically identifying the head, torso, and diaper of the infants. This is achieved by integrating a Faster R-CNN architecture, which includes a Region Proposal Network (RPN) and an object detection module. This allows the system to not only detect the presence of an infant but also to precisely locate and orient these specific body regions within the video frames.

Once these body parts are identified, the system extracts multiple photoplethysmographic imaging (PPGi) signals from each region. These raw PPGi signals are then processed through detrending and filtering stages to remove noise and artifacts. A crucial aspect is the signal quality assessment, which uses an activity index, Bayesian change point detection, and a beat-by-beat quality index (SQIbeat) to flag noisy or unreliable beats.

Heart rate estimation is performed using autoregressive modeling, which offers advantages over methods like FFT for short time-series data. To further improve robustness, the paper employs a data fusion approach. Heart rate estimates from individual body parts (head, torso) and the entire skin area are combined. This fusion is weighted based on the signal quality (SQI) and Kalman filter residuals, giving more importance to cleaner signals.

Datasets

The study utilized data from the Neonatal High-Dependency Unit (Neonatal HDU) study. This involved video recordings of 30 preterm infants in the NICU at the John Radcliffe Hospital, Oxford, UK. The recordings were captured under regular ambient lighting for up to four consecutive days. The data was divided into two sets:

  • Training set: 10 infants, 10 recording sessions.
  • Test set: 10 infants, 10 recording sessions.

The paper notes that the data was stored in a lossless, uncompressed format, with each hour of video generating over 408 GB of data, highlighting the storage and processing challenges.

Key Insights and Ideas

  • Localized signals are better: Extracting PPGi signals from specific body parts (head and torso) yields better results than using the entire skin area alone. This suggests that certain regions are less prone to motion artifacts or have more consistent blood flow signals.
  • Data fusion is key: Combining heart rate estimates from multiple regions (head, torso, and entire skin) significantly improves accuracy and increases the proportion of time for which reliable estimates can be obtained. This is a powerful demonstration of how combining complementary information can overcome individual signal limitations.
  • Body part detection is crucial for localization: The ability to accurately detect and segment specific body parts, even with rotation, is essential for extracting localized PPGi signals. This is a significant advancement over previous methods that relied on broader skin segmentation.
  • Signal quality assessment is paramount: The paper emphasizes the importance of robust signal quality assessment to filter out unreliable beats and improve the overall accuracy of heart rate estimation.

Key Results

  • Body Part Detection Performance: The CNN achieved high average precisions (AP) for body part detection: 98.2% for the head, 96.6% for the torso, and 75.2% for the diaper.
  • Heart Rate Estimation Accuracy: The combined approach (skin, head, and torso) achieved a mean absolute error (MAE) of 2.4 beats per minute for 80% of the time (31.1 hours) on the test set. This is a substantial improvement over using individual regions.
  • Estimated Time: The combined approach achieved the highest estimated time, covering 80.6% of the total recording time on the test set.
  • Comparison to Baseline: The errors from localized regions and fused estimates were generally lower than using the entire skin area as a baseline, and the proportion of estimated time was improved.

Limitations and Future Work

  • Diaper Detection: The network had some difficulties detecting the diaper when it was mostly covered by a blanket. While not critical for heart rate estimation, this information could be useful for respiratory rate estimation.
  • Real-time Performance: While the network can process video, there is room for improvement in achieving true real-time performance. The authors suggest optimizing the CNN architecture and object detection branch.
  • Motion Artifacts: The paper acknowledges that significant motion artifacts, particularly during patient movement, still lead to unreliable estimates and low signal quality.
  • Future Work: The authors propose several avenues for future research:
  • Improving the CNN model to reduce running time.
  • Incorporating more advanced computer vision techniques to enhance signal quality assessment.
  • Exploring Gaussian process regression to predict heart rate estimates during short periods where data is incomplete.

Summary updated

Abstract

Non-contact vital-sign estimation allows the monitoring of physiological parameters (such as heart rate, respiratory rate, and peripheral oxygen saturation) without contact electrodes or sensors. Our recent work has demonstrated that a convolutional neural network (CNN) can be used to detect the presence of a patient and segment the patient’s skin area for vital-sign estimation, thus enabling the automatic continuous monitoring of vital signs in a hospital environment. In a study approved by the local Research Ethical Committee, we made video recordings of pre-term infants nursed in a Neonatal Intensive Care Unit (NICU) at the John Radcliffe Hospital in Oxford, UK. We extended the CNN model to detect the head, torso and diaper of the infants. We extracted multiple photoplethysmographic imaging (PPGi) signals from each body part, analysed their signal quality, and compared them with the PPGi signal derived from the entire skin area. Our results demonstrated the benefits of estimating heart rate combined from multiple regions of interest using data fusion. In the test dataset, we achieved a mean absolute error of 2.4 beats per minute for 80% (31.1 hours) from a total recording time of 38.5 hours for which both reference heart rate and video data were valid.
BibTeX
@misc{green2018localised,
  author = {Gabrielle Green and Sitthichok Chaichulee and Mauricio Villarroel and João Jorge and Carlos Arteta and Andrew Zisserman and Lionel Tarassenko and Kenny McCormick},
  title = {{Localised photoplethysmography imaging for heart rate estimation of pre-term infants in the clinic}},
  year = {2018},
  doi = {10.1117/12.2289759},
}
Figure