Respiratory Rate Estimation from Thermal Video Data Using Spatio-Temporal Deep Learning

Mohsen Mozafari, Andrew Law, Rafik A. Goubran, James R. Green2024 · DOI

TL;DR

This paper introduces a novel deep learning approach for estimating respiratory rate (RR) from thermal video data, achieving state-of-the-art accuracy with a low error of 1.6 breaths per minute. By combining a face detection transformer with a spatio-temporal 3D CNN and bi-directional LSTM, the method effectively captures subtle breathing patterns while addressing the inherent time lag between thermal video and gold-standard respiratory signals.

What Problem Does This Paper Address?

Accurate and non-contact monitoring of vital signs, particularly respiration rate (RR), is crucial for remote health assessment and long-term patient care. Traditional methods, like respiratory effort belts, can be uncomfortable and restrictive. While video-based methods offer a non-contact alternative, they face challenges:

  • Privacy and Information Richness: Thermal video offers a privacy-preserving yet informative data source, capturing subtle temperature changes related to breathing.
  • Signal Extraction Complexity: Extracting a reliable RR signal from video requires sophisticated processing to isolate relevant physiological information from noise and irrelevant background.
  • ROI Detection: Automatically identifying the region of interest (ROI) for signal extraction is essential for a fully automated system.
  • Time Asynchrony: A significant challenge is the inherent time lag between the thermal signal (e.g., facial temperature changes) and gold-standard measurements (e.g., chest circumference changes), which can hinder traditional regression-based loss functions.
  • Limited Thermal Video Research: Much of the existing deep learning research in vital sign estimation has focused on RGB video and heart rate (HR) estimation, leaving a gap in robust RR estimation from thermal data.

Methods

The proposed method employs a multi-stage deep learning pipeline:

  1. Facial ROI Detection (Detection Transformer - DeTr): A DeTr model, fine-tuned on thermal facial data, is used to accurately detect the subject's facial region of interest in each thermal frame. This ensures that subsequent processing focuses on the most relevant area.
  2. Spatio-Temporal Feature Extraction (3D CNN): The dynamically cropped thermal video sequences are fed into a shallow 3D Convolutional Neural Network (CNN). This network is designed to capture short-term spatio-temporal patterns within the video, effectively processing the "width," "height," and "time" dimensions simultaneously.
  3. Long-Term Temporal Dependency Modeling (Bi-directional LSTM): Following the 3D CNN, a bi-directional Long Short-Term Memory (LSTM) layer is employed. This stage is crucial for capturing longer-term temporal dependencies in the respiratory signal, which is essential for accurate RR estimation. The bi-directional nature allows the model to learn from both forward and reverse temporal sequences within a given window.
  4. Signal Reconstruction and RR Estimation: A fully connected layer reconstructs the respiratory signal from the LSTM output. The final RR is then calculated by identifying the peak frequency within the reconstructed signal's frequency spectrum, after applying a bandpass filter.
  5. Novel Loss Function: To address the time asynchrony between thermal video and ground truth signals, a novel loss function is introduced. It combines:
  6. Negative Maximum Cross-Correlation (NMCC): This term accounts for the phase shift between the two signals.
  7. Absolute Frequency Peak Difference (AFD): This term ensures that the peak frequencies of the estimated and ground truth signals are similar, directly contributing to accurate RR estimation.

Datasets

  • Training and Evaluation Dataset: Data was collected from 22 participants (18 male/4 female) under four conditions: sitting with a mask, sitting without a mask, standing with a mask, and standing without a mask. Thermal videos were captured using a FLIR® T650sc camera at 30 fps.
  • Gold Standard Data: Simultaneous respiratory effort data was recorded using a SleepSense® respiratory effort sensor belt at 2048 Hz.
  • External Datasets (for pre-training/fine-tuning): The DeTr model was fine-tuned on the large TFW dataset (thermal images of human faces) and then further fine-tuned on a subset of the authors' own dataset.

Key Insights and Ideas

  • Thermal Video's Advantage: Thermal video is a promising modality for RR estimation due to its privacy-preserving nature and ability to capture subtle temperature variations linked to breathing.
  • Spatio-Temporal + Long-Term Modeling: Combining 3D CNNs for short-term spatio-temporal processing with bi-directional LSTMs for long-term temporal dependencies is key to robustly capturing respiratory patterns.
  • Addressing Time Lag is Crucial: The novel loss function, incorporating NMCC and AFD, is a critical innovation that directly tackles the phase shift problem, leading to significantly improved accuracy.
  • Masks Can Be Beneficial: Interestingly, wearing a mask can improve RR estimation performance by providing an additional surface for heat absorption from exhaled air.
  • Facial Hair Aids Detection: Subjects with beards showed better performance, suggesting that facial hair can act as a heat-absorbing surface, enhancing the thermal signal.
  • Movement Impacts Performance: Standing conditions resulted in slightly lower performance compared to sitting, likely due to increased body sway, highlighting the importance of controlled environments for optimal results.

Key Results

  • State-of-the-Art Accuracy: The proposed method achieved an average RR estimation error of 1.6 ± 0.4 breaths per minute, outperforming existing state-of-the-art methods.
  • High Accuracy Clips: 71% of the video clips resulted in an error of less than 2 BPM.
  • Fast Inference: The model achieves a runtime of 5.1 ms for a 10-second video window, enabling near real-time RR estimation.
  • Robustness to Masks: The model performed equally well with and without face masks.
  • Condition-Specific Performance: Seated subjects yielded more reliable results (error of 1.1-1.5 BPM) compared to standing subjects (error of 1.8-2.1 BPM).

Limitations and Future Work

  • Dataset Quality: The study relied on a high-quality dataset collected in a controlled environment with minimal movement and thermal interference.
  • Uncontrolled Environments: The model's performance may decrease in conditions with significant uncontrolled movements or thermal interference. The authors expect it to be most performant in controlled settings like hospital monitoring.
  • External Validation: The authors were unable to find a suitable external dataset with both thermal video and ground truth respiratory signals to validate their approach.
  • Future Work:
    • Multimodal Dataset: The authors plan to collect a multimodal video dataset (thermal, RGB, etc.) for public release.
  • Direct RR Regression: For applications where only RR is needed (not the full signal), modifying the model to directly estimate RR by emphasizing the MAE loss term and potentially simplifying the architecture is suggested.

Summary updated

BibTeX
@misc{mozafari2024respiratory,
  author = {Mohsen Mozafari and Andrew Law and Rafik A. Goubran and James R. Green},
  title = {{Respiratory Rate Estimation from Thermal Video Data Using Spatio-Temporal Deep Learning}},
  year = {2024},
  doi = {10.3390/s24196386},
}
Figure