TL;DR
This paper tackles the challenge of recognizing arousal levels from physiological data across diverse datasets. They introduce a novel deep learning ensemble method that combines knowledge from multiple sources, outperforming traditional machine learning and individual deep learning models. This work paves the way for more robust and generalizable emotion recognition systems by effectively leveraging heterogeneous data.
What Problem Does This Paper Address?
Affective computing, the field focused on building emotion-aware systems, has seen slow progress despite its potential in areas like human-computer interaction and healthcare. Arousal, representing the intensity of emotions, is a key dimension but remains difficult to accurately recognize from physiological signals. A major hurdle is the heterogeneity of publicly available datasets: they often use different sensors, data collection protocols, and labeling schemes, making it hard to train a single model that generalizes well. This paper aims to bridge this gap by developing a method that can effectively learn from and combine these diverse, inter-domain datasets for arousal recognition.
Methods
The core of this paper is a novel Deep Neural Network (DNN) ensemble method designed for inter-domain arousal recognition. The approach involves several key steps:
- Data Harmonization: Physiological signals (GSR, ECG, BVP) from different datasets are preprocessed into a common spectro-temporal space. This involves transforming ECG/BVP into R-R intervals and normalizing GSR data.
- Two DNN Architectures:
- DNN-Features: This approach extracts handcrafted features from the harmonized R-R and GSR data (e.g., heart rate variability features, GSR peak analysis) and feeds them into a fully connected DNN.
- CNN-GSR: This architecture directly uses the processed GSR signals as input to a Convolutional Neural Network (CNN). The CNN is designed to automatically learn relevant features from the raw temporal GSR data.
- DNN Meta-Learner: The outputs (predictions or learned representations) from both the DNN-Features and CNN-GSR models are then fed into a final fully connected DNN, termed the "meta-learner." This meta-learner combines the knowledge from the two specialized DNNs to make the final arousal prediction. This ensemble approach allows the model to benefit from both handcrafted features and automatically learned features.
Datasets
The study utilizes six publicly available datasets for arousal recognition:
- Ascertain: Labeled with a 7-point arousal scale.
- Deap: Labeled with a 9-point arousal scale.
- Driving workload dataset: Arousal inferred from workload ratings during driving.
- Cognitive load dataset: Labeled for subjective stress level during cognitive tasks.
- Mahnob: Labeled with subjective arousal ratings.
- Amigos: Labeled with subjective arousal ratings.
These datasets collectively provide 142 hours of arousal-labeled data from 191 subjects, captured using various sensors like GSR, ECG, and BVP. The paper highlights that while the data types are similar, variations in sensors and labeling scales necessitate the proposed harmonization and ensemble approach.
Key Insights and Ideas
- Leveraging Heterogeneity: The paper demonstrates that by harmonizing data and using an ensemble of DNNs, it's possible to effectively combine knowledge from diverse, even seemingly incompatible, datasets. This is crucial for building more robust models in real-world scenarios where data is rarely uniform.
- CNN-GSR's Feature Discovery: The CNN-GSR model, by learning directly from raw GSR signals, discovered that the peaks in the GSR signal are particularly informative for arousal recognition. This suggests that deep learning can uncover subtle, domain-specific patterns that might be missed by traditional feature engineering.
- Ensemble Power: The ensemble approach, combining specialized DNNs (one for features, one for raw GSR) with a meta-learner, significantly outperforms individual models and classical ML methods. This highlights the benefit of diverse model perspectives in tackling complex recognition tasks.
Key Results
The proposed DNN ensemble method achieved an average accuracy of 70% across the six datasets. This represents a significant improvement:
- 4 percentage points higher than other DNN-based methods (DNN-Features and CNN-GSR individually).
- At least 7 percentage points higher than classical machine learning methods.
On specific datasets, the improvement was even more pronounced. For instance, on the Mahnob dataset, the DNN-ensemble achieved 20 percentage points higher accuracy than other methods, and on the Amigos dataset, it achieved 10 percentage points higher accuracy. For the other four datasets, the ensemble performed comparably to the best individual methods.
Limitations and Future Work
- Limitations: The paper acknowledges that while the ensemble method shows promise, its performance on some datasets was similar to other methods. This suggests that further improvements might be needed for certain data characteristics or specific datasets. The preprocessing step of transforming data into a common space is a prerequisite and might introduce its own limitations or biases.
- Future Work: The authors plan to extend their approach to recognize the other two dimensions of emotion: valence and dominance. They also aim to explore the behavior of their DNNs on entirely new domains not included in the training phase, further testing the generalizability of their inter-domain learning strategy.
Summary updated