Temporal-spatial Representation Learning Transformer for EEG-based Emotion Recognition

Zhe Wang, Yongxiong Wang, Chuanfei Hu, Zhong Yin, Yu SongarXiv 2022 · DOI · arXiv

Abstract

Both the temporal dynamics and spatial correlations of Electroencephalogram (EEG), which contain discriminative emotion information, are essential for emotion recognition. However, redundant information within EEG signals often degrades performance, as subjects may only reach intense emotional states for a fraction of the stimulus duration. Furthermore, extracting discriminative features from complex spatial correlations among numerous electrodes remains a challenge. We propose the Temporal-Spatial EEG Representation Transformer (TSERT) to robustly capture these dynamics. TSERT utilizes weight-shared temporal feature extractors to adaptively extract context from raw signals and a multi-head self-attention mechanism to localize vital EEG fragments and essential brain regions. Experiments on the DEAP and MAHNOB-HCI datasets demonstrate that TSERT achieves state-of-the-art performance in arousal and valence classification.

Introduction

Recognizing human emotions through EEG signals is a critical goal for advancing human-computer interaction. However, EEG data is inherently high-dimensional and noisy. Two primary hurdles exist: temporal redundancy, where emotional intensity is transient during a stimulus, and spatial complexity, arising from the intricate correlations between electrodes across the scalp. Traditional methods often rely on handcrafted features like Power Spectral Density (PSD), which can obscure fine-grained temporal information vital for accurate recognition.

Methodology: The TSERT Architecture

The proposed TSERT model is an end-to-end Transformer-based architecture designed to address these challenges through two main stages:

  1. Weight-Shared Temporal Feature Extraction: Raw EEG signals are divided into "slices" or patches. These are processed by Transformer encoders that share weights across all channels. This weight-sharing acts as an adaptive filter, allowing the model to learn universal temporal patterns while reducing parameter count and preventing overfitting.
  2. Hierarchical Spatial Learning: To manage spatial complexity, TSERT employs a two-step process. It first analyzes electrode-level data within specific functional brain regions (e.g., frontal, occipital). These regional summaries are then integrated by a brain-region-level encoder to capture the global brain state.

The core of this architecture is the multi-head self-attention mechanism, which enables the model to automatically prioritize relevant time segments and highlight the most active brain regions during emotional responses.

The TSERT architecture, illustrating the flow from raw EEG signals through temporal and hierarchical spatial learning stages to final emotion classification.

Experimental Setup

The model was evaluated on two benchmark datasets:

  • DEAP: 32-channel EEG data from subjects watching music videos.
  • MAHNOB-HCI: Multimodal data from subjects watching movie clips.

A "leave-one-out" cross-subject validation strategy was employed. This is a rigorous test of the model's ability to generalize to unseen individuals, addressing the significant variability in brain activity across different people.

Results and Discussion

TSERT achieved state-of-the-art results in cross-subject classification. On the DEAP dataset, it reached 68.87% accuracy for arousal and 67.59% for valence. On the MAHNOB-HCI dataset, performance was even higher, with 70.02% accuracy for arousal and 69.32% for valence.

Performance comparison of TSERT against baseline models and ablation variants on the DEAP and MAHNOB-HCI datasets.

Key insights from the results include:

  • Raw Data Superiority: TSERT performed better on raw EEG signals than on traditional PSD features, suggesting that micro-fluctuations in brainwaves are crucial for distinguishing emotions.
  • Anatomy-Informed Design: The hierarchical spatial encoder's success confirms that mimicking the functional organization of the brain helps capture spatial correlations more effectively than flat neural networks.
  • Synergy of Time and Space: Comparison with variants like SERT (Spatial only) and TERT (Temporal only) shows that the integrated temporal-spatial approach is essential for peak performance.

Conclusion

TSERT represents a significant step forward in robust EEG-based emotion recognition. By leveraging the attention mechanisms of Transformers and a brain-inspired hierarchical structure, it effectively filters noise and focuses on discriminative signals. Future work will explore transfer learning and domain adaptation to further improve cross-subject generalization, making emotion-aware AI more viable for real-world applications such as medical diagnostics and assistive robotics.

Summary updated

Abstract

Both the temporal dynamics and spatial correlations of Electroencephalogram (EEG), which contain discriminative emotion information, are essential for the emotion recognition. However, some redundant information within the EEG signals would degrade the performance. Specifically,the subjects reach prospective intense emotions for only a fraction of the stimulus duration. Besides, it is a challenge to extract discriminative features from the complex spatial correlations among a number of electrodes. To deal with the problems, we propose a transformer-based model to robustly capture temporal dynamics and spatial correlations of EEG. Especially, temporal feature extractors which share the weight among all the EEG channels are designed to adaptively extract dynamic context information from raw signals. Furthermore, multi-head self-attention mechanism within the transformers could adaptively localize the vital EEG fragments and emphasize the essential brain regions which contribute to the performance. To verify the effectiveness of the proposed method, we conduct the experiments on two public datasets, DEAP and MAHNOBHCI. The results demonstrate that the proposed method achieves outstanding performance on arousal and valence classification.
BibTeX
@misc{wang2022temporal,
  author = {Zhe Wang and Yongxiong Wang and Chuanfei Hu and Zhong Yin and Yu Song},
  title = {{Temporal-spatial Representation Learning Transformer for EEG-based Emotion Recognition}},
  year = {2022},
  eprint = {2211.08880},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2211.08880},
}

Loading related content...

Figure