StressNet: Detecting Stress in Thermal Videos

Satish Kumar, A. S. M. Iftekhar, Michael Goebel, Tom Bullock, Mary H. MacLean, Michael B. Miller, Tyler Santander, Barry Giesbrecht, Scott T. Grafton, B. S. Manjunath

TL;DR

StressNet is a novel deep learning framework that detects human stress from thermal video by reconstructing a specific physiological marker called the Initial Systolic Time Interval (ISTI). By combining a physics-based heat emission model with a spatio-temporal neural network, the system achieves high-precision, non-contact stress monitoring that outperforms traditional metrics like heart rate and heart rate variability.

What Problem Does This Paper Address?

While remote health monitoring has advanced significantly, most current "contact-less" systems rely on RGB video to estimate heart rate (HR) or heart rate variability (HRV). However, HR and HRV are imperfect proxies for stress because they are influenced by both the sympathetic ("fight or flight") and parasympathetic ("rest and digest") nervous systems.

The authors target a more robust metric: the Initial Systolic Time Interval (ISTI). ISTI measures the time difference between the heart's electrical activation (ECG peak) and the actual mechanical opening of the aortic valve (measured via impedance cardiography, or ICG). ISTI is a pure index of cardiac sympathetic activity, making it a "gold standard" for quantifying acute stress. The challenge is that ISTI normally requires bulky, wired sensors (ECG + ICG). StressNet aims to estimate this signal using only non-invasive thermal video.

Methods

The StressNet architecture consists of four primary components:

  1. Emission Representation Model: Unlike RGB models that look at reflected light, this module models how heat is directly emitted and absorbed by the skin and underlying blood vessels. It uses a log non-linearity to suppress outliers and isolate the subtle physiological "pulses" from the static heat of the face.
  2. Spatial Module: The researchers used a ResNet-50 backbone (pre-trained on ImageNet) to extract high-level features from the processed thermal frames. They found that average pooling worked better than max pooling here, as it preserved subtle physiological signals that might otherwise be discarded as "noise."
  3. Temporal Module: A 6-layer LSTM (Long Short-Term Memory) network processes the spatial features. The input sequences were specifically tuned to 1-second intervals to match the typical human cardiac cycle, allowing the model to better capture the relationship between consecutive heartbeats.
  4. Detection & Classification: Instead of simple regression, the model uses a "binned" approach to predict ISTI values, which provides more stability against outliers. The predicted ISTI signal is then fed into a final classifier to determine the probability of the subject being under stress.

Datasets

The authors introduced a specialized dataset from the Biomarkers of Stress States (BOSS) study:

  • Participants: 42 healthy adults.
  • Protocol: Used the Cold Pressor Test (CPT) to induce stress (immersing feet in $0.5^\circ C$ water) and a Warm Pressor Test (WPT) as a control ($34^\circ C$ water).
  • Ground Truth: Simultaneous high-fidelity recordings of ECG and ICG were used to calculate the "true" ISTI values.
  • Video: High-resolution thermal video (Flir A655sc) recorded at 15 Hz.

Key Insights and Ideas

  • ISTI is Superior to HR: The most significant insight is that predicted ISTI (0.842 Average Precision) is a much better predictor of stress than even ground truth Heart Rate (0.753 AP) or HRV (0.814 AP). This validates the choice of ISTI as the target metric.
  • Thermal for Privacy: The authors highlight that thermal imaging is inherently more privacy-preserving than RGB video because it does not store a "true likeness" of the individual's face, making it more suitable for clinical or workplace monitoring.
  • Physics-Informed Preprocessing: By adapting the dichromatic reflection model (standard in computer vision) to the physics of thermal emission, the authors created a representation that is much more sensitive to blood flow changes than raw thermal pixels.

Key Results

  • Signal Estimation: StressNet achieved a Pearson Correlation Coefficient of 0.843 and a Mean Squared Error (MSE) of 5.845 when predicting the ISTI signal, significantly outperforming existing rPPG methods like DeepPhys.
  • Stress Detection: The model achieved an Average Precision (AP) of 0.842 in classifying stress vs. no-stress states.
  • Ablation Success: Removing the "Emission Representation" preprocessing step caused the error (MSE) to increase by 1.119, proving the importance of the physics-based input model.

Limitations and Future Work

  • Motion Sensitivity: The model struggles with large head movements or when a participant covers their face with their hands (common reactions to stress/pain).
  • Stress Diversity: The study focused on physical pain-induced stress. The authors note that future work should investigate if the model generalizes to social or mental stress.
  • Dataset Scale: While 42 participants is comparable to other rPPG studies, the authors suggest that a larger dataset would allow for training more complex spatial backbones from scratch rather than relying on ImageNet pre-training.
  • Environmental Factors: The experiments were conducted in a controlled lab setting; real-world atmospheric noise and varying distances from the camera remain challenges to be addressed.

Summary updated

Abstract

Precise measurement of physiological signals is critical for the effective monitoring of human vital signs. Recent developments in computer vision have demonstrated that signals such as pulse rate and respiration rate can be extracted from digital video of humans, increasing the possibility of contact-less monitoring. This paper presents a novel approach to obtaining physiological signals and classifying stress states from thermal video. The proposed network–"StressNet"–features a hybrid emission representation model that models the direct emission and absorption of heat by the skin and underlying blood vessels. This results in an information-rich feature representation of the face, which is used by spatio-temporal network for reconstructing the ISTI ( Initial Systolic Time Interval : a measure of change in cardiac sympathetic activity that is considered to be a quantitative index of stress in humans). The reconstructed ISTI signal is fed into a stress-detection model to detect and classify the individual’s stress state (i.e. stress or no stress). A detailed evaluation demonstrates that Stress-Net achieves estimated the ISTI signal with 95% accuracy and detect stress with average precision of 0.842.
BibTeX
@inproceedings{kumar2021stressnet,
  author = {Satish Kumar and A. S. M. Iftekhar and Michael Goebel and Tom Bullock and Mary H. MacLean and Michael B. Miller and Tyler Santander and Barry Giesbrecht and Scott T. Grafton and B. S. Manjunath},
  title = {{StressNet: Detecting Stress in Thermal Videos}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=qF7fBO4Rn_X},
}

Loading related content...

Figure