Introduction
Affective computing aims to detect emotional states, such as arousal (the level of physiological activation), using physiological signals like heart rate (ECG) and skin conductance (EDA) captured by wearable devices. Traditionally, this task relies on hand-crafted features, which often struggle to generalize across different hardware, sensor placements, or experimental protocols. While deep learning offers a path toward automated feature extraction, it typically requires vast amounts of human-labeled data—a resource that is both expensive and scarce in the field of physiological monitoring.
Furthermore, aggregating data from multiple studies (multi-corpus training) is notoriously difficult because different studies use different psychological scales to label emotions. This paper proposes an unsupervised, multi-modal machine learning framework that learns robust feature representations from unlabeled biosignals, enabling the effective use of diverse, aggregated datasets for state-of-the-art arousal detection.
Proposed Framework
The researchers developed a two-stage pipeline designed to minimize reliance on labeled data while maximizing the utility of multi-corpus information.
1. Unsupervised Representation Learning
The core of the framework consists of two separate stacked 1D convolutional autoencoders—one for ECG and one for EDA. After standardizing and segmenting the signals into 10-second windows, these autoencoders learn to compress the input data into a low-dimensional "latent" space and then reconstruct it. This process forces the network to identify the most salient structural features of the biosignals without needing explicit labels. L1 regularization is applied to enforce sparsity, ensuring the model captures only the most relevant information.
2. Feature Fusion and Classification
The autoencoders produce 80 latent features for each modality. These 160 features are concatenated (feature-level fusion) and fed into a Random Forest classifier. By decoupling feature extraction from classification, the supervised stage requires significantly less labeled data to achieve high performance in binary arousal classification (high vs. low).
Experimental Setup and Datasets
To evaluate generalizability, the authors aggregated four distinct datasets into a single multi-corpus repository. Arousal ratings were normalized to a 9-point scale and binarized (low < 5, high ≥ 6).
- AMIGOS: 40 subjects watching emotional videos.
- ASCERTAIN: 58 subjects watching movie clips.
- MAHNOB-HCI: 30 subjects viewing movie fragments and pictures.
- CLEAS: A dataset featuring 10 subjects undergoing high-stress medical simulations. This is the first time this dataset has been used for arousal classification.
Key Results and Insights
The framework consistently outperformed hand-crafted feature baselines and standard supervised Convolutional Neural Networks (CNNs) across all benchmarks:
- AMIGOS: 93% accuracy (vs. 89% previous best).
- ASCERTAIN: 79% accuracy (vs. 75% previous best).
- MAHNOB-HCI: 83% accuracy (vs. 82% previous best).
- CLEAS: 99% accuracy.
- Multi-Corpus Aggregate: 89% accuracy and 0.91 F1 score.
Critical Findings
- Data Aggregation Benefits: Unlike hand-crafted features, which often degrade when datasets are combined, the autoencoders thrived on the increased diversity of the multi-corpus data, learning more robust and generalizable features.
- Modality Importance: While EDA is traditionally considered the primary indicator for arousal, t-SNE visualizations of the latent space revealed that learned ECG features were more discriminative and separable.
- Fusion Strategy: Feature-level "early" fusion significantly outperformed decision-level "late" fusion, suggesting that the interaction between ECG and EDA features contains vital information for emotion recognition.
Conclusion and Future Directions
This work demonstrates that unsupervised representation learning can bridge the gap between small, disparate physiological datasets and the data-hungry requirements of deep learning. By learning from the inherent structure of biosignals, the model achieves superior generalizability across different experimental contexts. Future research will explore the use of Recurrent Neural Networks (RNNs) to capture temporal dynamics and extend the framework to continuous emotion estimation rather than binary classification.
Summary updated