Deep-Learning-Based Stress Recognition with Spatial-Temporal Facial Information

Taejae Jeon, Han Byeol Bae, Yongju Lee, Sungjun Jang, Sangyoun LeeSensors 2021 · OpenReview

TL;DR

This paper introduces a novel deep-learning approach for stress recognition using facial information, combining spatial and temporal attention mechanisms with facial landmark data. They also introduce a new, large-scale facial image database for stress recognition, addressing a critical data gap and achieving state-of-the-art performance on their benchmark.

What Problem Does This Paper Address?

Recognizing and managing stress is crucial for well-being, but real-time, accurate stress detection remains a challenge. Existing methods often rely on physiological signals, which require cumbersome wearable devices. While facial image-based methods avoid this, they traditionally depend on handcrafted features that lack discriminative power. Deep learning offers a path to learn more robust features, but requires substantial, well-annotated datasets, which are scarce for stress recognition. This paper aims to bridge this gap by developing a deep-learning model that leverages rich facial information and by creating a comprehensive dataset to train and evaluate it.

Methods

The core of the proposed method is a deep-learning architecture that integrates three key components:

  • Spatial Attention Module: This module learns to focus on specific, stress-relevant regions within a facial image, assigning higher weights to these areas.
  • Facial Landmark Feature Module: Recognizing that subtle facial movements (eyes, mouth, head) are indicative of stress, this module extracts features from facial landmarks. These landmarks are preprocessed (min-max normalization and Gaussian blurring) to mitigate jittering from detection algorithms.
  • Temporal Attention Module: For video input, this module assigns higher weights to frames that are most indicative of stress over time, capturing dynamic changes in facial expressions.

These modules are combined within a ResNet-18 backbone, and the entire network is trained end-to-end using a custom loss function designed to handle the specific characteristics of their stress dataset (inter-subject variability vs. intra-subject subtlety).

Datasets

The authors introduce the Yonsei Stress Image Database, a large-capacity dataset specifically created for stress recognition research. This database consists of:

  • Subjects: 50 individuals (20-39 years old).
  • Data: Video recordings of subjects performing tasks designed to induce different stress levels (neutral, low stress, high stress). These tasks included reading scripts and interviews in native and non-native languages.
  • Annotation: Stress levels are labeled according to the experimental scenario.
  • Size: Over 2 million images, captured at 24 frames per second, with facial regions cropped and resized to 112x112 pixels for processing.

This dataset is publicly available on IEEE DataPort.

Key Insights and Ideas

  • Facial Landmarks are Crucial: The inclusion of facial landmark features significantly boosted performance, highlighting their importance in capturing subtle stress-related facial movements that might be missed by image-based features alone.
  • Attention is Key for Spatio-Temporal Dynamics: Both spatial and temporal attention modules proved vital. Spatial attention helps pinpoint critical facial regions, while temporal attention captures the crucial dynamic evolution of stress expressions over time.
  • Dataset Design Matters: The creation of a large, well-annotated dataset with controlled stress-inducing scenarios was fundamental to enabling effective deep-learning for this task.
  • Custom Loss Function for Data Characteristics: The proposed loss function, combining cross-entropy with a margin-based MSE component, effectively addresses the challenge of high inter-subject variability and low intra-subject stress-related changes.

Key Results

  • The proposed method achieved the highest accuracy of 66.8409% on their newly constructed database, outperforming various established deep-learning models and handcrafted feature-based methods.
  • Even without the facial landmark feature, the proposed model achieved 65.3396% accuracy, demonstrating the strength of the spatial-temporal attention mechanism.
  • The inclusion of facial landmark features improved accuracy across all stress states, particularly for the high-stress category.
  • Experiments showed that a clip length of 2 seconds (48 frames) yielded the best performance, balancing temporal information with computational feasibility.

Limitations and Future Work

  • Subtle Stress Recognition: The authors acknowledge that recognizing very subtle stress changes, especially in individuals with less pronounced facial expressions, remains challenging.
  • Difficult Environments: Future work will focus on improving performance in more challenging environments, such as those with facial occlusions.
  • Multimodal Stress Recognition: To address the limitations of relying solely on visual cues, future research will explore multimodal approaches, incorporating voice data, which is closely related to stress.

Summary updated

Abstract

In recent times, as interest in stress control has increased, many studies on stress recognition have been conducted. Several studies have been based on physiological signals, but the disadvantage of this strategy is that it requires physiological-signal-acquisition devices. Another strategy employs facial-image-based stress-recognition methods, which do not require devices, but predominantly use handcrafted features. However, such features have low discriminating power. We propose a deep-learning-based stress-recognition method using facial images to address these challenges. Given that deep-learning methods require extensive data, we constructed a large-capacity image database for stress recognition. Furthermore, we used temporal attention, which assigns a high weight to frames that are highly related to stress, as well as spatial attention, which assigns a high weight to regions that are highly related to stress. By adding a network that inputs the facial landmark information closely related to stress, we supplemented the network that receives only facial images as the input. Experimental results on our newly constructed database indicated that the proposed method outperforms contemporary deep-learning-based recognition methods.
BibTeX
@inproceedings{jeon2021deep,
  author = {Taejae Jeon and Han Byeol Bae and Yongju Lee and Sungjun Jang and Sangyoun Lee},
  title = {{Deep-Learning-Based Stress Recognition with Spatial-Temporal Facial Information}},
  booktitle = {Sensors},
  year = {2021},
  url = {https://openreview.net/forum?id=Kv25ZxZY4H},
}
Figure