Contactless Estimation of Respiratory Frequency Using 3D-CNN on Thermal Images

Federica Gioia, Filippo Pura, Alberto Greco, Dario Piga, Arcangelo Merla, Marco ForgionearXiv 2026 · DOI · arXiv

This paper presents a significant step toward making contactless health monitoring more practical. Researchers from the University of Pisa and SUPSI have developed a deep learning approach to estimate respiratory frequency ($f_R$) directly from thermal video, eliminating the need for the complex manual tracking and pre-processing that has long hindered this technology.

TL;DR

The authors developed a 3D Convolutional Neural Network (3D-CNN) that estimates respiratory rate directly from raw thermal video without manual region-of-interest tracking. By combining sim-to-real transfer learning with data augmentation, the model achieves robust performance on small clinical datasets, paving the way for non-intrusive monitoring in cars and hospitals.

What Problem Does This Paper Address?

Respiratory rate is a "neglected" vital sign—critical for diagnosing everything from pneumonia to cardiac arrest, yet often ignored because traditional sensors (like chest belts or nasal thermistors) are uncomfortable or invasive.

Infrared Thermography (IRT) is a promising contactless alternative because it can "see" the temperature changes caused by breathing: exhaled air is warmer than inhaled air. However, current IRT methods usually require sophisticated algorithms to identify and track specific regions of interest (ROIs), like the nostrils, across every frame. If a subject moves their head or the video has low contrast, these methods often fail. The authors aimed to create an "end-to-end" system that works directly on raw video data, bypassing these brittle pre-processing steps.

Methods

The core of the solution is a 3D Convolutional Neural Network (3D-CNN). Unlike standard 2D-CNNs used for photos, a 3D-CNN processes a "cube" of data, allowing it to capture spatial features (the shape of the face) and temporal features (the rhythmic change in temperature over time) simultaneously.

To train a deep model on a small dataset (only 30 subjects), the authors employed two clever strategies:

  1. Sim-to-Real Transfer Learning: They created a massive synthetic dataset by taking static thermal face images and overlaying pulsating circles on the nostril areas to simulate breathing. The model first learned the "basics" of breathing from this artificial data.
  2. Data Augmentation: They used the RandAugment algorithm to introduce noise, blurring, and rotations into the training data, forcing the model to become more robust to real-world imperfections.
Figure 2: Overview of the experimental setup. The participant was seated in a thermally con- trolled room, facing a FLIR T640 thermal camera positioned at a 1-meter distance. The thermal camera acquired infrared facial images, while a chest belt connected to a BIOPAC MP150 sys- tem recorded the reference respiratory signal. The setup ensured synchronized acquisition of both contactless and ground-truth respiratory data.

Datasets

  • Real-world Dataset: 30 healthy volunteers were recorded using a high-end FLIR thermal camera. To vary their breathing rates, they performed a baseline rest, a cognitive stress test (Stroop test), and an emotional elicitation task using images from the IAPS database.
  • Synthetic Dataset: An arbitrarily large dataset generated by combining fixed facial backgrounds with simulated, randomized respiratory activity.

Key Insights and Ideas

  • Automatic Focus: Visualization of the network's internal activations (Figure 11 in the paper) revealed that the trained model naturally "learned" to focus on the nostril region, even though it was never explicitly told to look there.
  • The Synergy of Synthetic Data: Interestingly, data augmentation alone did not improve performance. It was only when combined with synthetic pre-training that the model's accuracy jumped significantly. This suggests that for niche medical tasks, "teaching" the model the underlying physics via simulation is vital.
  • Raw Data Robustness: One of the most important findings was that the model performed nearly identically on raw thermal video as it did on pre-processed video (where eyes and backgrounds were manually masked). This proves that deep learning can handle the "messiness" of real-world data that usually breaks traditional algorithms.
Figure 1: Experimental timeline showing the succession of 5-min tasks administered to each subject. After an initial acclimatization period (dashed line), participants underwent a baseline resting phase, followed by a Stroop test (cognitive stress induction), a second resting period, and an emotional elicitation task using affective images.

Key Results

The final model, using the combination of transfer learning and augmentation, achieved:

  • A validation $R^2$ score of 0.604 on raw thermal videos and 0.614 on pre-processed videos.
  • A Mean Squared Error (MSE) of approximately 0.30, indicating high precision in estimating breaths per minute.
  • The model successfully tracked respiratory trends across different subjects and stress levels, as shown in the comparison between predicted and true values.

Limitations and Future Work

While promising, the study has a few hurdles to clear before it hits the real world:

  • Motion Sensitivity: The current synthetic data didn't include significant head movements. In a real car or a neonatal unit, subjects move, which might degrade performance.
  • Frequency Range: The experiments only covered a range of 9 to 30 breaths per minute. Future work needs to test the model on higher frequencies, such as those seen during intense physical activity or medical emergencies.
  • Hardware Costs: The study used a high-quality, expensive thermal camera. The authors suggest that testing the model on low-cost thermal sensors is a critical next step for widespread adoption.

Summary updated

Abstract

Monitoring physiological parameters such as respiratory rate ($f_R$) is essential for diagnosing and managing various pathological conditions. Thermal imaging offers a promising contactless alternative to traditional methods, which often rely on partially invasive sensors or obtrusive wearable systems. However, existing approaches for $f_R$ estimation from thermal signals typically require extensive pre-processing and manual or semi-automatic region-of-interest (ROI) tracking, limiting their practical applicability. This study proposes a deep learning-based method for estimating $f_R$ directly from thermal videos, eliminating the need for complex pre-processing and ROI tracking. A 3D Convolutional Neural Network (3D-CNN) is developed to operate on raw thermal video data. To address challenges related to small datasets, the model is trained using data augmentation and transfer learning from synthetic datasets. Experimental results demonstrate that the proposed approach achieves a validation $R^2$ score of approximately 0.61 on both pre-processed and raw thermal videos. By simplifying the workflow, this method holds promise for enhancing the feasibility of thermal imaging in real-world applications, such as remote healthcare and driver monitoring in automotive applications.
BibTeX
@misc{gioia2026contactless,
  author = {Federica Gioia and Filippo Pura and Alberto Greco and Dario Piga and Arcangelo Merla and Marco Forgione},
  title = {{Contactless Estimation of Respiratory Frequency Using 3D-CNN on Thermal Images}},
  year = {2026},
  eprint = {2611.26367},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.26367},
}
Figure