Joint Use of a Low Thermal Resolution Thermal Camera and an RGB Camera for Respiration Measurement

Zaeed Khan, Matias Rusanen, Miika Arvonen, Timo Leppänen, Simo Särkkä

TL;DR

Researchers have developed a hybrid system that pairs a high-resolution smartphone RGB camera with a low-cost, low-resolution thermal camera to monitor respiration. By using the RGB camera to track facial features and mapping that data onto the thermal feed, the system can accurately identify complex breathing patterns and count breath cycles without the need for bulky, expensive medical-grade thermal hardware.

What Problem Does This Paper Address?

Monitoring respiration rate (RR) and breathing patterns is vital for diagnosing everything from sleep apnea to lung disease. While thermal imaging is an ideal non-contact method—it detects the temperature changes at the nostrils during inhalation and exhalation—it usually requires high-resolution cameras (e.g., $320 \times 240$ or $640 \times 480$). These cameras are expensive, bulky, and not portable.

Low-resolution thermal cameras (like the FLIR One Pro used here, at $160 \times 120$) are affordable and smartphone-compatible, but they have a major flaw: the image is too "blurry" for computer vision algorithms to reliably find and track nostrils, especially if the subject moves. This paper tackles the challenge of using these "low-fidelity" sensors to produce "high-fidelity" respiratory data.

Methods

The authors propose a multi-stage pipeline that leverages the strengths of different sensors:

  1. The Triple-Camera Setup: The system uses a smartphone's main RGB camera, the FLIR One Pro’s internal RGB camera, and the FLIR’s thermal sensor.
  2. Cross-Spectrum Mapping: Since the thermal camera can't track nostrils well, the high-res smartphone RGB camera does the heavy lifting. It uses a Single Shot Multibox Detector (SSD) and ResNet-10 architecture to find the face and nostrils. This Region of Interest (ROI) is then mathematically mapped onto the thermal frame using a two-step calibration process.
  3. Synchronization via Sound: To align the video feeds from different devices, the authors used sound analysis. Participants said "calibration begins," allowing the researchers to sync the frames based on the audio peaks.
  4. Signal Processing: Once the thermal signal is extracted (by averaging pixel intensities in the nostril ROI), it passes through a pipeline of standardization, median filtering (to remove spikes), interpolation, and a high-pass Butterworth filter to isolate the breathing frequency.
  5. Validation: The results were compared against a "gold standard" reference: Respiratory Inductance Plethysmography (RIP) belts worn by the subjects.

Datasets

The study involved a small-scale experimental dataset:

  • Participants: 7 healthy male volunteers (data from 6 was used for final analysis), aged 24–46.
  • Protocols: Participants performed four specific tasks: normal breathing (still), normal breathing (with head movement), deep inhalations with the tongue pressed to the palate (to test nose-wing movement), and a "Breath Dictator" protocol where they followed a video simulating clinical patterns like apnea (stopped breathing), tachypnea (rapid breathing), and Kussmaul breathing (deep, labored breathing).

Key Insights and Ideas

  • RGB as a "Guide" for Thermal: The core "aha" moment is that you don't need a high-res thermal sensor if you have a high-res RGB sensor to act as a spatial guide. By "slaving" the thermal ROI to the RGB tracker, the system maintains focus on the nostrils even when the thermal image looks like a featureless blob.
  • Pattern over Rate: Interestingly, the system was better at identifying the shape and frequency changes of breathing (patterns) than it was at pinpointing the exact instantaneous breathing rate. This suggests the tool might be more useful for diagnosing irregular breathing types than just simple counting.
  • The Android Jitter: The researchers noted that Android smartphones have unstable frame rates, which makes time-synchronization a significant hurdle for multi-camera systems. Their use of audio cues was a pragmatic workaround for this hardware limitation.

Key Results

  • Breathing Cycles: The system achieved a decent correlation ($R^2 = 0.59$) for counting the total number of breathing cycles.
  • Pattern Accuracy: For the complex "Breath Dictator" patterns, the Mean Absolute Error (MAE) was $0.1235\text{ Hz}$ and the Root Mean Square Error (RMSE) was $0.1684\text{ Hz}$. These results actually outperformed some previous studies that used higher-resolution thermal cameras.
  • Movement Sensitivity: The system performed well during still protocols but saw a significant performance drop when participants moved their heads freely, highlighting that cross-spectrum mapping is highly sensitive to alignment.

Limitations and Future Work

The primary limitation is motion. If a subject turns their head significantly, the tracker in the RGB video can lose the ROI, or the mapping to the thermal frame becomes misaligned. Additionally, the calibration process currently requires manual selection of points, which isn't practical for a consumer-ready "plug-and-play" device.

Future work aims to automate the calibration process and improve the tracking algorithms to handle more robust head movements. The authors also suggest that using a single sound source to sync all devices automatically would make the system much more user-friendly.

Summary updated

Abstract

Recently, joint use of thermal camera and RGB camera has emerged as a potential solution for respiration measurement. Bulky and large thermal cameras with high thermal resolution have been mostly used in thermal-imaging-based respiration measurement studies. We aim to present a method for respiration measurement using an RGB camera and a low thermal resolution thermal camera. Thermal cameras with low thermal resolution are small in size and easily portable, allowing flexible and a simple implementation of the measurement system. First, the cameras were calibrated and synchronized using calibration objects and sound analysis tools, respectively. Second, a cross-spectrum mapping method was developed which was followed by the extraction of the respiration signal. Third, the extracted and reference respiration signals were aligned. In the last step of the proposed method, a signal processing framework was applied to the estimated and reference respiration signals. The performance of the cross-spectrum mapping method was visually evaluated. The mapping method performed well when a participant had little to no head movements, however, the performance decreased when head movements were present. Our analysis showed that there is a relatively good agreement between the proposed and the reference respiration measurement methods. The number of breathing cycles from the estimated respiration signal was computed with a decent accuracy. However, the accuracy of measuring breathing rate is low compared to the previous studies. The estimated breathing patterns were measured with better accuracy than in the previous work. This study demonstrates that respiration measurement can be done by the joint use of an RGB camera and thermal camera, that has a low thermal resolution, with a decent level of accuracy.
BibTeX
@article{khan2023joint,
  author = {Zaeed Khan and Matias Rusanen and Miika Arvonen and Timo Leppänen and Simo Särkkä},
  title = {{Joint Use of a Low Thermal Resolution Thermal Camera and an RGB Camera for Respiration Measurement}},
  journal = {IEEE Transactions on Instrumentation and Measurement},
  year = {2023},
  doi = {10.1109/tim.2023.3334345},
}
Figure