Abstract
Both the temporal dynamics and spatial correlations of Electroencephalogram (EEG), which contain discriminative emotion information, are essential for emotion recognition. However, redundant information within EEG signals often degrades performance, as subjects may only reach intense emotional states for a fraction of the stimulus duration. Furthermore, extracting discriminative features from complex spatial correlations among numerous electrodes remains a challenge. We propose the Temporal-Spatial EEG Representation Transformer (TSERT) to robustly capture these dynamics. TSERT utilizes weight-shared temporal feature extractors to adaptively extract context from raw signals and a multi-head self-attention mechanism to localize vital EEG fragments and essential brain regions. Experiments on the DEAP and MAHNOB-HCI datasets demonstrate that TSERT achieves state-of-the-art performance in arousal and valence classification.
Introduction
Recognizing human emotions through EEG signals is a critical goal for advancing human-computer interaction. However, EEG data is inherently high-dimensional and noisy. Two primary hurdles exist: temporal redundancy, where emotional intensity is transient during a stimulus, and spatial complexity, arising from the intricate correlations between electrodes across the scalp. Traditional methods often rely on handcrafted features like Power Spectral Density (PSD), which can obscure fine-grained temporal information vital for accurate recognition.
Methodology: The TSERT Architecture
The proposed TSERT model is an end-to-end Transformer-based architecture designed to address these challenges through two main stages:
- Weight-Shared Temporal Feature Extraction: Raw EEG signals are divided into "slices" or patches. These are processed by Transformer encoders that share weights across all channels. This weight-sharing acts as an adaptive filter, allowing the model to learn universal temporal patterns while reducing parameter count and preventing overfitting.
- Hierarchical Spatial Learning: To manage spatial complexity, TSERT employs a two-step process. It first analyzes electrode-level data within specific functional brain regions (e.g., frontal, occipital). These regional summaries are then integrated by a brain-region-level encoder to capture the global brain state.
The core of this architecture is the multi-head self-attention mechanism, which enables the model to automatically prioritize relevant time segments and highlight the most active brain regions during emotional responses.
Experimental Setup
The model was evaluated on two benchmark datasets:
- DEAP: 32-channel EEG data from subjects watching music videos.
- MAHNOB-HCI: Multimodal data from subjects watching movie clips.
A "leave-one-out" cross-subject validation strategy was employed. This is a rigorous test of the model's ability to generalize to unseen individuals, addressing the significant variability in brain activity across different people.
Results and Discussion
TSERT achieved state-of-the-art results in cross-subject classification. On the DEAP dataset, it reached 68.87% accuracy for arousal and 67.59% for valence. On the MAHNOB-HCI dataset, performance was even higher, with 70.02% accuracy for arousal and 69.32% for valence.
Key insights from the results include:
- Raw Data Superiority: TSERT performed better on raw EEG signals than on traditional PSD features, suggesting that micro-fluctuations in brainwaves are crucial for distinguishing emotions.
- Anatomy-Informed Design: The hierarchical spatial encoder's success confirms that mimicking the functional organization of the brain helps capture spatial correlations more effectively than flat neural networks.
- Synergy of Time and Space: Comparison with variants like SERT (Spatial only) and TERT (Temporal only) shows that the integrated temporal-spatial approach is essential for peak performance.
Conclusion
TSERT represents a significant step forward in robust EEG-based emotion recognition. By leveraging the attention mechanisms of Transformers and a brain-inspired hierarchical structure, it effectively filters noise and focuses on discriminative signals. Future work will explore transfer learning and domain adaptation to further improve cross-subject generalization, making emotion-aware AI more viable for real-world applications such as medical diagnostics and assistive robotics.
Summary updated