Enhancing Generalization in PPG-Based Emotion Measurement with a CNN-TCN-LSTM Model

Karim Alghoul, Hussein Al Osman, Abdulmotaleb El-Saddik

Abstract

In the field of affective computing, recognizing human emotions through physiological signals remains a significant challenge due to the high degree of inter-subject variability. While Photoplethysmogram (PPG) signals—commonly captured by wearable devices—offer a non-invasive and objective window into emotional states, existing models often fail to generalize to new individuals. This paper presents a hybrid deep-learning architecture combining Convolutional Neural Networks (CNN), Long Short-Term Memory (LSTM) networks, and Temporal Convolutional Networks (TCN) to bridge this "generalization gap." By leveraging the complementary strengths of these architectures, the proposed model achieves superior performance in classifying valence and arousal across unseen subjects.

Introduction

Affective computing aims to develop systems capable of interpreting and responding to human emotional states. Traditional modalities, such as facial expressions or vocal analysis, are susceptible to social masking and external manipulation. In contrast, physiological signals like the Photoplethysmogram (PPG)—which measures blood volume changes via optical sensors in smartwatches—provide a more robust and "unmaskable" metric of the autonomic nervous system's response to stimuli.

However, the primary hurdle in PPG-based emotion recognition is generalization. Most models exhibit high accuracy when tested on the same individuals used during training but suffer significant performance degradation when applied to new users. This is largely due to the unique physiological signatures each person possesses. To address this, researchers at the University of Ottawa have developed a multi-branch hybrid model designed to extract universal, robust features from noisy PPG data.

Methodology

The proposed architecture utilizes a "triple-threat" approach, processing raw PPG signals through three specialized neural stages to capture different dimensions of the data.

1. Spatial Feature Extraction (CNN)

The first stage employs a Convolutional Neural Network (CNN) to act as a primary feature extractor. The CNN analyzes the "spatial" morphology of the PPG waveform, identifying characteristics such as pulse amplitude and specific wave shapes that correlate with emotional shifts.

2. Temporal Dynamics (LSTM)

To account for the time-dependent nature of heart rate data, the model incorporates Long Short-Term Memory (LSTM) layers. LSTMs are specifically designed to handle sequences, allowing the model to capture the sequential dependencies and temporal dynamics of the heart rate as it evolves over the course of an emotional experience.

3. Hierarchical Context (TCN)

The "secret sauce" of the architecture is the Temporal Convolutional Network (TCN). TCNs excel at capturing hierarchical patterns in time-series data without the risk of "information leakage" from future time steps into the past. By running the TCN and LSTM branches in parallel, the model can simultaneously process immediate temporal changes and broader hierarchical context.

Finally, the outputs from these branches are concatenated and passed to a classification layer to predict two primary dimensions of emotion: Valence (the positivity or negativity of an emotion) and Arousal (the intensity of the emotion).

Experimental Setup

The model was evaluated using the PPGE (Photoplethysmogram Dataset for Emotional Analysis), which includes data from 18 participants (13 men, 5 women). The dataset consists of 72 PPG signals recorded at 100 Hz while participants viewed five-minute emotional stimuli.

To rigorously test the model's ability to generalize, the authors employed Leave-One-Subject-Out (LOSOCV) cross-validation. In this method, the model is trained on 17 subjects and tested on the 18th, repeating the process for every participant. This approach provides a more realistic assessment of how the model would perform in real-world applications compared to standard random data splits.

Results and Discussion

The study prioritized the Area Under the Curve (AUC) as the primary performance metric, as raw accuracy can be misleading in imbalanced datasets where certain emotional states are overrepresented. The hybrid CNN-TCN-LSTM model consistently outperformed standalone architectures in generalization tasks:

  • Valence Classification: The hybrid model achieved an AUC of 0.66, surpassing the baseline CNN (0.64) and CNN-LSTM (0.60).
  • Arousal Classification: The model achieved an AUC of 0.69, a significant improvement over the standalone CNN (0.61).

These results suggest that the synergy between TCN and LSTM is vital. While LSTMs are effective for short-term sequences, they often struggle with long-range hierarchical structures that TCNs handle with ease. By combining them, the model maintains a stable and reliable performance even when faced with the physiological quirks of a new subject.

Conclusion and Future Work

This research demonstrates that a hybrid deep-learning approach can significantly mitigate the generalization gap in PPG-based emotion recognition. While the current study is limited by a relatively small dataset of 18 participants and the inherent sensitivity of PPG to motion artifacts, it lays the groundwork for more robust affective computing systems.

Future research will focus on multi-modal fusion—integrating PPG with ECG (electrocardiogram) and EEG (brain waves)—and expanding the model to recognize specific multi-class emotions such as joy, frustration, or boredom. Furthermore, testing the model on data from wearables during daily activities will be essential to transition this technology from controlled lab environments to real-world health and safety applications.

Summary updated

BibTeX
@inproceedings{alghoul2025enhancing,
  author = {Karim Alghoul and Hussein Al Osman and Abdulmotaleb El-Saddik},
  title = {{Enhancing Generalization in PPG-Based Emotion Measurement with a CNN-TCN-LSTM Model}},
  booktitle = {I2MTC},
  year = {2025},
  doi = {10.1109/i2mtc62753.2025.11079085},
}
Figure