Multimodal Deep Learning for Remote Stress Estimation Using CCT-LSTM

Sayyedjavad Ziaratnia, Tipporn Laohakangvalvit, Midori Sugaya, Peeraya SripianWACV 2024

TL;DR

This paper introduces a novel multimodal deep learning framework, CCT-LSTM, that combines facial expressions and remote photoplethysmography (rPPG) to accurately estimate and classify stress levels from video data. By integrating Compact Convolutional Transformers (CCT) for feature extraction and Long Short-Term Memory (LSTM) for temporal analysis, the model achieves state-of-the-art results in both stress task and multilevel stress classification, offering a more robust and non-invasive approach to stress monitoring.

What Problem Does This Paper Address?

Stress is a pervasive issue with significant implications for physical and mental health, driving safety, and human-robot interaction. Early detection and monitoring of stress are crucial for mitigating its negative effects. Traditional methods like questionnaires and clinical interviews are often impractical for continuous monitoring. While behavioral and physiological signal analysis offer non-invasive alternatives, they have limitations. Behavioral analysis can be consciously controlled, and physiological methods often require direct physical contact and specialized equipment. Recent advancements in remote photoplethysmography (rPPG) have opened doors for non-invasive stress detection, but accurately classifying stress tasks and, more importantly, differentiating between multiple levels of stress (e.g., relaxation vs. low stress vs. high stress) remains a significant challenge. Existing multimodal approaches have shown promise, but there's a need for more sophisticated architectures that can effectively fuse information from different modalities and capture temporal dynamics for precise stress estimation.

Methods

The core of this research is the CCT-LSTM pipeline, a novel multimodal deep learning architecture designed for remote stress estimation. The approach leverages two key deep learning techniques:

  • Compact Convolutional Transformer (CCT): This component is used for feature extraction from both facial landmarks and rPPG signals. CCT is chosen for its efficiency in handling spatial information (crucial for facial landmarks) and its ability to manage complex relationships between features, while being more computationally efficient and data-light than standard Vision Transformers (ViT).
  • Long Short-Term Memory (LSTM): Following feature extraction, LSTM networks are employed to recognize temporal patterns within the extracted features. LSTMs are well-suited for time-series data, enabling the model to capture the dynamic nature of stress markers.

The multimodal aspect is achieved by processing facial expressions (using 478 landmarks) and rPPG signals independently through CCT, and then feeding these extracted features into a shared LSTM layer for fusion and temporal analysis. This bidirectional use of the CCT-LSTM framework for each modality enhances the robustness and accuracy of the stress estimation.

For data preprocessing, facial landmarks are extracted using MediaPipe, and then transformed into image representations using the Markov Transition Field (MTF) method to capture temporal dependencies. Similarly, rPPG signals are extracted and also converted into MTF images.

Datasets

The study primarily utilizes the UBFC-Phys dataset [22]. This dataset was specifically curated for psycho-physiological research and includes data from 56 subjects. It contains facial videos and biosignals (including rPPG and EDA) collected under three different tasks designed to induce varying stress levels:

  • T1 (Relaxation): A baseline, low-stress condition.
  • T2 (Speech Task): A moderate stress task.
  • T3 (Arithmetic Task): A challenging task designed to induce higher stress.

The dataset also includes two groups of participants: a control group (facing easier tasks) and a test group (facing more challenging tasks), allowing for stress level classification. The authors note that this dataset was used for both stress task classification (T1, T2, T3) and multilevel stress classification (T1, T3-ctrl, T3-test).

Key Insights and Ideas

  • Multimodality is Key for Robust Stress Estimation: The research strongly suggests that combining facial expressions and rPPG signals significantly outperforms using either modality in isolation for stress detection. This highlights the complementary nature of these physiological and behavioral indicators.
  • CCT-LSTM Architecture Synergizes Strengths: The integration of CCT for efficient, spatially-aware feature extraction and LSTM for capturing temporal dynamics proves to be a powerful combination for analyzing complex, time-varying physiological signals related to stress.
  • MTF as an Effective Image Representation for Time-Series: The use of the Markov Transition Field (MTF) to convert time-series data (facial landmarks and rPPG) into image representations allows for the application of powerful computer vision models like CCT, effectively bridging the gap between time-series analysis and image-based deep learning.
  • Multilevel Stress Classification is a Significant Challenge: While stress task classification is relatively well-addressed, accurately distinguishing between different levels of stress (e.g., low vs. high stress) is considerably more difficult and requires more sophisticated models.

Key Results

The proposed CCT-LSTM model demonstrated state-of-the-art performance on the UBFC-Phys dataset:

  • Stress Task Classification (T1, T2, T3): The model achieved a mean accuracy of 83.2% and an F1 score of 83.4% in sevenfold cross-validation for the three-class task classification. This significantly outperformed previous methods, including those by Sabour et al. and Zhang et al.
  • Multilevel Stress Classification (T1, T3-ctrl, T3-test): For differentiating between relaxation, lower stress (T3-ctrl), and higher stress (T3-test), the multimodal CCT-LSTM pipeline achieved a mean accuracy of 80.5% and an F1 score of 80.4% in fivefold cross-validation. This represents a substantial improvement over standalone modalities (CCT-rPPG: 60.6% accuracy, CCT-Facial Landmark: 62.0% accuracy).
  • Comparison to Baselines: Table 1 shows the CCT-LSTM model outperforming various other methods like MLP, LIT, DFAF, CAM, MFN, and BCSA in stress task classification. Table 2 and 3 show its superiority in multilevel stress classification compared to individual modalities.

Limitations and Future Work

  • Dataset Size: The authors acknowledge that the variation in F1 scores across different folds in the multilevel stress classification experiment suggests that the dataset might not have a sufficient number of samples. This could be a potential limitation affecting the model's generalization.
  • Hyperparameter Tuning: Future work will involve an in-depth analysis of specific hyperparameters, such as window size and moving interval, during data preprocessing.
  • Time-Series to Image Conversion: Investigating alternative techniques for converting time-series data into images is another area for future exploration. These efforts aim to further refine the methodologies and potentially enhance stress detection accuracy.

Summary updated

BibTeX
@inproceedings{ziaratnia2024multimodal,
  author = {Sayyedjavad Ziaratnia and Tipporn Laohakangvalvit and Midori Sugaya and Peeraya Sripian},
  title = {{Multimodal Deep Learning for Remote Stress Estimation Using CCT-LSTM}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure