Introduction
Recognizing human emotion is a cornerstone of advanced human-computer interaction. While audio-visual cues are common, they are susceptible to intentional masking. Physiological signals, such as electroencephalograms (EEG) and electrocardiograms (ECG), offer a more objective "gold standard" for emotion recognition as they are difficult to fake. However, effectively utilizing these signals requires overcoming two primary challenges: capturing the synergy between spatial, temporal, and spectral dimensions, and managing the inherent heterogeneity of different sensor types (e.g., brain waves vs. heart rate).
Methodology
The proposed HetEmotionNet addresses these challenges through a two-stream heterogeneous graph recurrent neural network architecture designed to handle the complexity of multi-modal physiological data.
Graph-Based Representation
Traditional methods often map sensor data onto 2D grids, which can introduce noise through zero-filling and artificial interpolation. HetEmotionNet instead represents sensors as nodes in a graph, preserving their natural topological relationships. Edges between these nodes are defined using Mutual Information, capturing how much information one sensor shares with another regardless of their physical distance.
Two-Stream Architecture
The framework processes data through two parallel paths to capture complementary emotional cues across different domains:
- Spatial-Temporal Stream: Analyzes raw signal amplitudes to track how brain and body activity evolves over time.
- Spatial-Spectral Stream: Examines Differential Entropy (DE) features across various frequency bands (theta, alpha, beta, gamma) to identify the "rhythm" of the emotional response.
The Triple-Threat Layer
Each stream utilizes a sequence of three specialized components to extract deep features:
- Graph Transformer Network (GTN): This component handles heterogeneity by learning "meta-paths" to understand the distinct relationships between different signal types, such as an EEG node and an ECG node.
- Graph Convolutional Network (GCN): This layer captures spatial correlations and functional connectivity between different regions of the body and brain.
- Gated Recurrent Unit (GRU): This handles the recurrent aspect, capturing dependencies over time (in the temporal stream) or across frequency bands (in the spectral stream).
Datasets and Evaluation
The model was validated on two widely recognized benchmarks in the affective computing community:
- DEAP: Contains data from 32 participants watching 40 music videos, including 32-channel EEG and 8 peripheral physiological signals (EOG, EMG, GSR, etc.).
- MAHNOB-HCI: Features 27 participants watching 20 video clips, providing 32-channel EEG and 6 peripheral signals (ECG, GSR, respiration, etc.).
Results and Discussion
HetEmotionNet established new benchmarks for both datasets, significantly outperforming previous state-of-the-art models like ACRNN and SST-EmotionNet.
- DEAP Dataset: Achieved 97.66% accuracy for Valence (pleasantness) and 97.30% for Arousal (intensity).
- MAHNOB-HCI Dataset: Achieved 93.95% for Valence and 93.90% for Arousal.
Ablation studies confirmed that the inclusion of the GTN and the dual-stream approach were critical to these results. Notably, the model also showed the lowest standard deviation among all tested methods, indicating that its performance is highly stable across different subjects and trials.
Conclusion
HetEmotionNet demonstrates the power of treating multi-modal physiological data as a dynamic, heterogeneous graph. By leveraging both temporal and spectral information, it provides a robust framework for objective emotion recognition. Future research will focus on reducing computational complexity for real-time edge applications and improving cross-subject generalization through transfer learning.
Summary updated