TL;DR
Researchers have developed a hybrid system that pairs a high-resolution smartphone RGB camera with a low-cost, low-resolution thermal camera to monitor respiration. By using the RGB camera to track facial features and mapping that data onto the thermal feed, the system can accurately identify complex breathing patterns and count breath cycles without the need for bulky, expensive medical-grade thermal hardware.
What Problem Does This Paper Address?
Monitoring respiration rate (RR) and breathing patterns is vital for diagnosing everything from sleep apnea to lung disease. While thermal imaging is an ideal non-contact method—it detects the temperature changes at the nostrils during inhalation and exhalation—it usually requires high-resolution cameras (e.g., $320 \times 240$ or $640 \times 480$). These cameras are expensive, bulky, and not portable.
Low-resolution thermal cameras (like the FLIR One Pro used here, at $160 \times 120$) are affordable and smartphone-compatible, but they have a major flaw: the image is too "blurry" for computer vision algorithms to reliably find and track nostrils, especially if the subject moves. This paper tackles the challenge of using these "low-fidelity" sensors to produce "high-fidelity" respiratory data.
Methods
The authors propose a multi-stage pipeline that leverages the strengths of different sensors:
- The Triple-Camera Setup: The system uses a smartphone's main RGB camera, the FLIR One Pro’s internal RGB camera, and the FLIR’s thermal sensor.
- Cross-Spectrum Mapping: Since the thermal camera can't track nostrils well, the high-res smartphone RGB camera does the heavy lifting. It uses a Single Shot Multibox Detector (SSD) and ResNet-10 architecture to find the face and nostrils. This Region of Interest (ROI) is then mathematically mapped onto the thermal frame using a two-step calibration process.
- Synchronization via Sound: To align the video feeds from different devices, the authors used sound analysis. Participants said "calibration begins," allowing the researchers to sync the frames based on the audio peaks.
- Signal Processing: Once the thermal signal is extracted (by averaging pixel intensities in the nostril ROI), it passes through a pipeline of standardization, median filtering (to remove spikes), interpolation, and a high-pass Butterworth filter to isolate the breathing frequency.
- Validation: The results were compared against a "gold standard" reference: Respiratory Inductance Plethysmography (RIP) belts worn by the subjects.
Datasets
The study involved a small-scale experimental dataset:
- Participants: 7 healthy male volunteers (data from 6 was used for final analysis), aged 24–46.
- Protocols: Participants performed four specific tasks: normal breathing (still), normal breathing (with head movement), deep inhalations with the tongue pressed to the palate (to test nose-wing movement), and a "Breath Dictator" protocol where they followed a video simulating clinical patterns like apnea (stopped breathing), tachypnea (rapid breathing), and Kussmaul breathing (deep, labored breathing).
Key Insights and Ideas
- RGB as a "Guide" for Thermal: The core "aha" moment is that you don't need a high-res thermal sensor if you have a high-res RGB sensor to act as a spatial guide. By "slaving" the thermal ROI to the RGB tracker, the system maintains focus on the nostrils even when the thermal image looks like a featureless blob.
- Pattern over Rate: Interestingly, the system was better at identifying the shape and frequency changes of breathing (patterns) than it was at pinpointing the exact instantaneous breathing rate. This suggests the tool might be more useful for diagnosing irregular breathing types than just simple counting.
- The Android Jitter: The researchers noted that Android smartphones have unstable frame rates, which makes time-synchronization a significant hurdle for multi-camera systems. Their use of audio cues was a pragmatic workaround for this hardware limitation.
Key Results
- Breathing Cycles: The system achieved a decent correlation ($R^2 = 0.59$) for counting the total number of breathing cycles.
- Pattern Accuracy: For the complex "Breath Dictator" patterns, the Mean Absolute Error (MAE) was $0.1235\text{ Hz}$ and the Root Mean Square Error (RMSE) was $0.1684\text{ Hz}$. These results actually outperformed some previous studies that used higher-resolution thermal cameras.
- Movement Sensitivity: The system performed well during still protocols but saw a significant performance drop when participants moved their heads freely, highlighting that cross-spectrum mapping is highly sensitive to alignment.
Limitations and Future Work
The primary limitation is motion. If a subject turns their head significantly, the tracker in the RGB video can lose the ROI, or the mapping to the thermal frame becomes misaligned. Additionally, the calibration process currently requires manual selection of points, which isn't practical for a consumer-ready "plug-and-play" device.
Future work aims to automate the calibration process and improve the tracking algorithms to handle more robust head movements. The authors also suggest that using a single sound source to sync all devices automatically would make the system much more user-friendly.
Summary updated