This paper presents a significant step toward making contactless health monitoring more practical. Researchers from the University of Pisa and SUPSI have developed a deep learning approach to estimate respiratory frequency ($f_R$) directly from thermal video, eliminating the need for the complex manual tracking and pre-processing that has long hindered this technology.
TL;DR
The authors developed a 3D Convolutional Neural Network (3D-CNN) that estimates respiratory rate directly from raw thermal video without manual region-of-interest tracking. By combining sim-to-real transfer learning with data augmentation, the model achieves robust performance on small clinical datasets, paving the way for non-intrusive monitoring in cars and hospitals.
What Problem Does This Paper Address?
Respiratory rate is a "neglected" vital sign—critical for diagnosing everything from pneumonia to cardiac arrest, yet often ignored because traditional sensors (like chest belts or nasal thermistors) are uncomfortable or invasive.
Infrared Thermography (IRT) is a promising contactless alternative because it can "see" the temperature changes caused by breathing: exhaled air is warmer than inhaled air. However, current IRT methods usually require sophisticated algorithms to identify and track specific regions of interest (ROIs), like the nostrils, across every frame. If a subject moves their head or the video has low contrast, these methods often fail. The authors aimed to create an "end-to-end" system that works directly on raw video data, bypassing these brittle pre-processing steps.
Methods
The core of the solution is a 3D Convolutional Neural Network (3D-CNN). Unlike standard 2D-CNNs used for photos, a 3D-CNN processes a "cube" of data, allowing it to capture spatial features (the shape of the face) and temporal features (the rhythmic change in temperature over time) simultaneously.
To train a deep model on a small dataset (only 30 subjects), the authors employed two clever strategies:
- Sim-to-Real Transfer Learning: They created a massive synthetic dataset by taking static thermal face images and overlaying pulsating circles on the nostril areas to simulate breathing. The model first learned the "basics" of breathing from this artificial data.
- Data Augmentation: They used the RandAugment algorithm to introduce noise, blurring, and rotations into the training data, forcing the model to become more robust to real-world imperfections.
Datasets
- Real-world Dataset: 30 healthy volunteers were recorded using a high-end FLIR thermal camera. To vary their breathing rates, they performed a baseline rest, a cognitive stress test (Stroop test), and an emotional elicitation task using images from the IAPS database.
- Synthetic Dataset: An arbitrarily large dataset generated by combining fixed facial backgrounds with simulated, randomized respiratory activity.
Key Insights and Ideas
- Automatic Focus: Visualization of the network's internal activations (Figure 11 in the paper) revealed that the trained model naturally "learned" to focus on the nostril region, even though it was never explicitly told to look there.
- The Synergy of Synthetic Data: Interestingly, data augmentation alone did not improve performance. It was only when combined with synthetic pre-training that the model's accuracy jumped significantly. This suggests that for niche medical tasks, "teaching" the model the underlying physics via simulation is vital.
- Raw Data Robustness: One of the most important findings was that the model performed nearly identically on raw thermal video as it did on pre-processed video (where eyes and backgrounds were manually masked). This proves that deep learning can handle the "messiness" of real-world data that usually breaks traditional algorithms.
Key Results
The final model, using the combination of transfer learning and augmentation, achieved:
- A validation $R^2$ score of 0.604 on raw thermal videos and 0.614 on pre-processed videos.
- A Mean Squared Error (MSE) of approximately 0.30, indicating high precision in estimating breaths per minute.
- The model successfully tracked respiratory trends across different subjects and stress levels, as shown in the comparison between predicted and true values.
Limitations and Future Work
While promising, the study has a few hurdles to clear before it hits the real world:
- Motion Sensitivity: The current synthetic data didn't include significant head movements. In a real car or a neonatal unit, subjects move, which might degrade performance.
- Frequency Range: The experiments only covered a range of 9 to 30 breaths per minute. Future work needs to test the model on higher frequencies, such as those seen during intense physical activity or medical emergencies.
- Hardware Costs: The study used a high-quality, expensive thermal camera. The authors suggest that testing the model on low-cost thermal sensors is a critical next step for widespread adoption.
Summary updated