Unsupervised Multi-Modal Representation Learning for Affective Computing with Multi-Corpus Wearable Data

Kyle Ross, Paul Hungler, Ali Etemad

Introduction

Affective computing aims to detect emotional states, such as arousal (the level of physiological activation), using physiological signals like heart rate (ECG) and skin conductance (EDA) captured by wearable devices. Traditionally, this task relies on hand-crafted features, which often struggle to generalize across different hardware, sensor placements, or experimental protocols. While deep learning offers a path toward automated feature extraction, it typically requires vast amounts of human-labeled data—a resource that is both expensive and scarce in the field of physiological monitoring.

Furthermore, aggregating data from multiple studies (multi-corpus training) is notoriously difficult because different studies use different psychological scales to label emotions. This paper proposes an unsupervised, multi-modal machine learning framework that learns robust feature representations from unlabeled biosignals, enabling the effective use of diverse, aggregated datasets for state-of-the-art arousal detection.

Proposed Framework

The researchers developed a two-stage pipeline designed to minimize reliance on labeled data while maximizing the utility of multi-corpus information.

The proposed multi-modal unsupervised representation learning pipeline.

1. Unsupervised Representation Learning

The core of the framework consists of two separate stacked 1D convolutional autoencoders—one for ECG and one for EDA. After standardizing and segmenting the signals into 10-second windows, these autoencoders learn to compress the input data into a low-dimensional "latent" space and then reconstruct it. This process forces the network to identify the most salient structural features of the biosignals without needing explicit labels. L1 regularization is applied to enforce sparsity, ensuring the model captures only the most relevant information.

2. Feature Fusion and Classification

The autoencoders produce 80 latent features for each modality. These 160 features are concatenated (feature-level fusion) and fed into a Random Forest classifier. By decoupling feature extraction from classification, the supervised stage requires significantly less labeled data to achieve high performance in binary arousal classification (high vs. low).

Experimental Setup and Datasets

To evaluate generalizability, the authors aggregated four distinct datasets into a single multi-corpus repository. Arousal ratings were normalized to a 9-point scale and binarized (low < 5, high ≥ 6).

Examples of ECG (a, b) and EDA (c, d) signals used in the study.
  • AMIGOS: 40 subjects watching emotional videos.
  • ASCERTAIN: 58 subjects watching movie clips.
  • MAHNOB-HCI: 30 subjects viewing movie fragments and pictures.
  • CLEAS: A dataset featuring 10 subjects undergoing high-stress medical simulations. This is the first time this dataset has been used for arousal classification.

Key Results and Insights

The framework consistently outperformed hand-crafted feature baselines and standard supervised Convolutional Neural Networks (CNNs) across all benchmarks:

  • AMIGOS: 93% accuracy (vs. 89% previous best).
  • ASCERTAIN: 79% accuracy (vs. 75% previous best).
  • MAHNOB-HCI: 83% accuracy (vs. 82% previous best).
  • CLEAS: 99% accuracy.
  • Multi-Corpus Aggregate: 89% accuracy and 0.91 F1 score.

Critical Findings

  • Data Aggregation Benefits: Unlike hand-crafted features, which often degrade when datasets are combined, the autoencoders thrived on the increased diversity of the multi-corpus data, learning more robust and generalizable features.
  • Modality Importance: While EDA is traditionally considered the primary indicator for arousal, t-SNE visualizations of the latent space revealed that learned ECG features were more discriminative and separable.
  • Fusion Strategy: Feature-level "early" fusion significantly outperformed decision-level "late" fusion, suggesting that the interaction between ECG and EDA features contains vital information for emotion recognition.

Conclusion and Future Directions

This work demonstrates that unsupervised representation learning can bridge the gap between small, disparate physiological datasets and the data-hungry requirements of deep learning. By learning from the inherent structure of biosignals, the model achieves superior generalizability across different experimental contexts. Future research will explore the use of Recurrent Neural Networks (RNNs) to capture temporal dynamics and extend the framework to continuous emotion estimation rather than binary classification.

Summary updated

Abstract

With recent developments in smart technologies, there has been a growing focus on the use of artificial intelligence and machine learning for affective computing to further enhance the user experience through emotion recognition. Typically, machine learning models used for affective computing are trained using manually extracted features from biological signals. Such features may not generalize well for large datasets and may be sub-optimal in capturing the information from the raw input data. One approach to address this issue is to use fully supervised deep learning methods to learn latent representations of the biosignals. However, this method requires human supervision to label the data, which may be unavailable or difficult to obtain. In this work we propose an unsupervised framework reduce the reliance on human supervision. The proposed framework utilizes two stacked convolutional autoencoders to learn latent representations from wearable electrocardiogram (ECG) and electrodermal activity (EDA) signals. These representations are utilized within a random forest model for binary arousal classification. This approach reduces human supervision and enables the aggregation of datasets allowing for higher generalizability. To validate this framework, an aggregated dataset comprised of the AMIGOS, ASCERTAIN, CLEAS, and MAHNOB-HCI datasets is created. The results of our proposed method are compared with using convolutional neural networks, as well as methods that employ manual extraction of hand-crafted features. The methodology used for fusing the two modalities is also investigated. Lastly, we show that our method outperforms current state-of-the-art results that have performed arousal detection on the same datasets using ECG and EDA biosignals. The results show the wide-spread applicability for stacked convolutional autoencoders to be used with machine learning for affective computing.
BibTeX
@article{ross2020unsupervised,
  author = {Kyle Ross and Paul Hungler and Ali Etemad},
  title = {{Unsupervised Multi-Modal Representation Learning for Affective Computing with Multi-Corpus Wearable Data}},
  journal = {J. Ambient Intell. Humaniz. Comput.},
  year = {2020},
  doi = {10.1007/s12652-021-03462-9},
}
Figure