Abstract
Integrating prior knowledge of neurophysiology into neural network architecture enhances the performance of emotion decoding. While numerous techniques emphasize learning spatial and short-term temporal patterns, there has been limited emphasis on capturing the vital long-term contextual information associated with emotional cognitive processes. In order to address this discrepancy, we introduce a novel transformer model called emotion transformer (EmT). EmT is designed to excel in both generalized cross-subject EEG emotion classification and regression tasks. In EmT, EEG signals are transformed into a temporal graph format, creating a sequence of EEG feature graphs using a temporal graph construction module (TGC). A novel residual multi-view pyramid GCN module (RMPG) is then proposed to learn dynamic graph representations for each EEG feature graph within the series, and the learned representations of each graph are fused into one token. Furthermore, we design a temporal contextual transformer module (TCT) with two types of token mixers to learn the temporal contextual information. Finally, the task-specific output module (TSO) generates the desired outputs. Experiments on four publicly available datasets show that EmT achieves higher results than the baseline methods for both EEG emotion classification and regression tasks.
Introduction
Decoding emotions from brain activity (EEG) is a complex challenge primarily due to significant cross-subject variability. Emotions are high-order cognitive processes that evolve over time and involve intricate, multi-region interactions. Existing models often fall short by relying on simplistic spatial modeling (static graphs), ignoring long-term temporal context, or lacking flexibility between classification and regression tasks. The Emotion Transformer (EmT) addresses these gaps by combining a "pyramid" of graph neural networks for spatial modeling with task-specific Transformer modules for temporal context.
Methods
The EmT architecture consists of four primary stages designed to capture both spatial and temporal dynamics of EEG signals.
- Temporal Graph Construction (TGC): Raw EEG signals are segmented and transformed into a sequence of graphs where electrodes serve as nodes. Features are extracted using relative Power Spectral Density (rPSD) across seven frequency bands.
- Residual Multi-view Pyramid GCN (RMPG): This module acts as the spatial engine. It utilizes parallel Graph Convolutional Networks (GCNs) with varying depths and learnable adjacency matrices. This "pyramid" structure enables the model to learn multiple views of brain connectivity, from local clusters to global networks.
- Temporal Contextual Transformer (TCT): To capture long-term dependencies, the TCT uses task-specific token mixers:
- TCT-Clas: Employs Multi-head Self-Attention (MSA) and Short-Time Aggregation (STA) to smooth noisy labels and focus on emotion-correlated signal components.
- TCT-Regr: Utilizes Recurrent Neural Networks (RNNs/GRUs) to better preserve the continuous sequential flow required for regression tasks.
- Task-specific Output Module (TSO): MLP heads generate the final classification categories or continuous regression values.
Results and Discussion
The model was evaluated on four public datasets: SEED, THU-EP, and FACED for classification, and MAHNOB-HCI for regression.
Key findings include:
- Classification Performance: EmT-Deep (8-layer) achieved a state-of-the-art accuracy of 80.2% on the SEED dataset. It also demonstrated strong generalization on the large-scale FACED dataset (123 subjects), suggesting it scales well to diverse populations.
- Regression Performance: On the MAHNOB-HCI dataset, the model achieved a PCC of 0.490 and a CCC of 0.396, significantly outperforming standard LSTM and TCN approaches.
- Architectural Insights: The RMPG module's ability to learn dynamic "brain maps" was crucial for performance. Additionally, while attention is effective for classification, recurrence (RNNs) proved superior for continuous emotion tracking in regression tasks, as it better preserves fine-grained temporal evolution.
Conclusion
EmT represents a significant advancement in cross-subject EEG emotion recognition by effectively integrating spatial pyramid GCNs with task-specific temporal transformers. While the model exhibits higher computational complexity than simpler CNNs, its superior accuracy and generalization capabilities make it a promising candidate for real-world Brain-Computer Interface (BCI) applications. Future work will focus on incorporating Riemannian manifolds for better geometric modeling and optimizing the architecture for real-time deployment.
Summary updated