StressNet: Detecting Stress in Thermal Videos

Satish Kumar, A. S. M. Iftekhar, Michael Goebel, Tom Bullock, Mary H. MacLean, Michael B. Miller, Tyler Santander, Barry Giesbrecht, Scott T. Grafton, B. S. Manjunath

Abstract

Precise measurement of physiological signals is critical for effective vital sign monitoring. Recent computer vision advances allow extracting pulse and respiration rates from digital video, enabling contactless monitoring. This paper introduces "StressNet," a novel approach for obtaining physiological signals and classifying stress states from thermal video. StressNet utilizes a hybrid emission representation model to capture heat emission and absorption by skin and blood vessels. This rich feature set enables a spatio-temporal network to reconstruct the Initial Systolic Time Interval (ISTI), a quantitative index of cardiac sympathetic activity and stress. The reconstructed ISTI signal is then used to classify stress states. Evaluation shows StressNet estimates ISTI with 95% accuracy and detects stress with an average precision of 0.842.

Introduction: The Challenge of Remote Stress Monitoring

Remote health monitoring has progressed significantly, yet most contactless systems rely on RGB video to estimate heart rate (HR) or heart rate variability (HRV). These metrics are imperfect proxies for stress as they are influenced by both the sympathetic and parasympathetic nervous systems.

This research targets the Initial Systolic Time Interval (ISTI). ISTI measures the time difference between the heart's electrical activation (ECG R-peak) and the mechanical opening of the aortic valve (B-point in impedance cardiography). ISTI serves as a "gold standard" for quantifying acute stress because it is a pure index of cardiac sympathetic activity. Traditionally, measuring ISTI requires bulky, wired sensors (ECG and ICG). StressNet aims to estimate this signal non-invasively using only thermal video.

Figure 1: Definition of ISTI using ECG and dZ/dt (ICG) signals.

Methods: The StressNet Architecture

StressNet is a deep learning framework designed to process thermal video and output a stress probability. The architecture consists of four primary components:

  1. Emission Representation Model: Unlike models for RGB video that rely on reflected light, this module models direct heat emission and absorption. It employs a log non-linearity to isolate subtle physiological pulses from static facial heat.
  2. Spatial Module: A ResNet-50 backbone, pre-trained on ImageNet, extracts high-level features from thermal frames. Average pooling is utilized to preserve subtle signals that might be lost with max pooling.
  3. Temporal Module: A 6-layer LSTM network processes spatial features in 1-second intervals to match the human cardiac cycle, capturing the relationship between consecutive heartbeats.
  4. Detection & Classification: The model uses a binned approach to predict ISTI values for stability. The resulting signal is fed into a classifier to determine the probability of stress.
Figure 2: The StressNet architecture overview.

Experimental Setup and Dataset

The researchers developed a specialized dataset from the Biomarkers of Stress States (BOSS) study involving 42 healthy adults. Stress was induced using the Cold Pressor Test (CPT), involving foot immersion in $0.5^\circ C$ water, with a Warm Pressor Test ($34^\circ C$) as a control. Ground truth ISTI values were calculated from simultaneous high-fidelity ECG and ICG recordings. High-resolution thermal video was captured at 15 Hz.

Key Results and Insights

StressNet demonstrates that predicted ISTI is a superior predictor of stress compared to traditional metrics.

  • Signal Accuracy: The model achieved a Pearson Correlation Coefficient of 0.843 and a Mean Squared Error (MSE) of 5.845 for ISTI prediction, outperforming existing methods like DeepPhys.
  • Stress Detection: The system achieved an Average Precision (AP) of 0.842 in classifying stress vs. no-stress states. Notably, ground truth HR and HRV only reached 0.753 AP and 0.814 AP, respectively, validating ISTI as a more robust metric.
  • Physics-Informed Preprocessing: Ablation studies showed that removing the Emission Representation step increased MSE by 1.119, highlighting the importance of modeling thermal physics.

Discussion and Future Work

While StressNet offers a privacy-preserving (thermal imaging does not store a "true likeness") and effective means of monitoring stress, limitations remain. The model is sensitive to large head movements and facial obstructions. Future research will explore generalization to social or mental stress, larger datasets to reduce reliance on pre-trained backbones, and performance in less controlled, real-world environments.

Summary updated

Abstract

Precise measurement of physiological signals is critical for the effective monitoring of human vital signs. Recent developments in computer vision have demonstrated that signals such as pulse rate and respiration rate can be extracted from digital video of humans, increasing the possibility of contact-less monitoring. This paper presents a novel approach to obtaining physiological signals and classifying stress states from thermal video. The proposed network--"StressNet"--features a hybrid emission representation model that models the direct emission and absorption of heat by the skin and underlying blood vessels. This results in an information-rich feature representation of the face, which is used by spatio-temporal network for reconstructing the ISTI ( Initial Systolic Time Interval: a measure of change in cardiac sympathetic activity that is considered to be a quantitative index of stress in humans ). The reconstructed ISTI signal is fed into a stress-detection model to detect and classify the individual's stress state ( i.e. stress or no stress ). A detailed evaluation demonstrates that StressNet achieves estimated the ISTI signal with 95% accuracy and detect stress with average precision of 0.842. The source code is available on Github.
BibTeX
@inproceedings{kumar2021stressnet,
  author = {Satish Kumar and A. S. M. Iftekhar and Michael Goebel and Tom Bullock and Mary H. MacLean and Michael B. Miller and Tyler Santander and Barry Giesbrecht and Scott T. Grafton and B. S. Manjunath},
  title = {{StressNet: Detecting Stress in Thermal Videos}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=qF7fBO4Rn_X},
}
Figure