MSBiLSTM-Attention: EEG Emotion Recognition Model Based on Spatiotemporal Feature Fusion

Yahong Ma, Zhentao Huang, Yuyao Yang, Zuowen Chen, Qi Dong, Shanwen Zhang, Yuan-Fang LiBiomimetics 2025 · DOI

TL;DR

This paper introduces MSBiLSTM-Attention, a novel deep learning model that fuses spatiotemporal features from raw EEG data for highly accurate emotion recognition. By combining multi-scale convolutional networks, bidirectional LSTMs, and an attention mechanism, the model effectively captures complex emotional states, achieving state-of-the-art performance on benchmark datasets.

What Problem Does This Paper Address?

Recognizing human emotions is crucial for advancing human-computer interaction, but accurately interpreting emotional states, especially through physiological signals like Electroencephalography (EEG), remains a significant challenge. EEG signals are complex, exhibiting both spatial patterns across different brain regions and temporal dynamics over time. Extracting these spatiotemporal features effectively and designing robust classifiers are key hurdles. While deep learning offers automatic feature extraction, many existing methods still rely on manual preprocessing or struggle to capture the intricate interplay of spatial and temporal information, leading to limitations in accuracy and generalization. This paper aims to address these challenges by developing a unified deep learning framework that automatically extracts and fuses spatiotemporal EEG features for precise emotion classification.

Methods

The proposed MSBiLSTM-Attention model is a sophisticated deep learning architecture designed to tackle the complexities of EEG-based emotion recognition. It operates on raw EEG data, eliminating the need for manual preprocessing. The core of the model comprises two parallel branches:

  • Multi-scale Convolutional Neural Network (CNN) (Block 1): This branch utilizes convolutional filters of varying sizes to capture spatial features from the EEG signals. Smaller filters focus on fine-grained local details, while larger filters capture broader contextual information, allowing for a comprehensive understanding of spatial patterns across different brain regions.
  • Bidirectional Long Short-Term Memory (Bi-LSTM) (Block 2): This branch processes the temporal dynamics of the EEG signals. Bi-LSTM is chosen over standard LSTM because it can learn dependencies from both past and future sequences, effectively capturing long-term temporal patterns and mitigating issues like gradient vanishing common in traditional recurrent networks.

These two branches extract spatiotemporal features independently. Subsequently, these features are concatenated and fed into an Attention Mechanism. This mechanism dynamically weighs the importance of different features, allowing the model to focus on the most relevant spatiotemporal information for accurate emotion classification. Finally, the attended features are processed through two fully connected layers and a Softmax classifier to predict the emotional state.

Datasets

The study utilizes two publicly available EEG datasets for emotion recognition:

  • SEED: This dataset contains EEG recordings from 15 participants, with each participant experiencing 15 trials involving positive, neutral, and negative emotion-inducing movie clips.
  • SEED-IV: This dataset also includes EEG recordings from 15 subjects but features 72 movie clips designed to evoke happiness, sadness, fear, or neutrality.

The paper notes that the data and category labels within these datasets are uniformly distributed and balanced. For their experiments, they extracted 1000 continuous recordings from each participant's video segment.

Key Insights and Ideas

  • End-to-End Processing of Raw EEG: By directly feeding raw EEG data into the model, the authors simplify the pipeline and reduce reliance on manual feature engineering, a common bottleneck in previous approaches.
  • Synergistic Spatiotemporal Feature Fusion: The model's strength lies in its ability to simultaneously extract and fuse spatial features (via multi-scale CNNs) and temporal features (via Bi-LSTMs). This integrated approach is crucial for capturing the nuanced nature of emotional responses reflected in EEG.
  • Attention for Feature Prioritization: The attention mechanism acts as a smart filter, enabling the model to automatically identify and prioritize the most informative electrode channels and features, thereby improving classification accuracy and robustness. This addresses the challenge of selecting relevant features manually.
  • Ablation Studies Highlight Component Importance: The ablation experiments clearly demonstrate that neither the spatial (Block1) nor the temporal (Block2) component alone can achieve the performance of the full MSBiLSTM-Attention model. This underscores the necessity of their combined and fused approach.

Key Results

The MSBiLSTM-Attention model demonstrated exceptional performance on both datasets:

  • SEED Dataset (3-class classification):
    • Single validation accuracy: 99.44%
    • 10-fold cross-validation accuracy: 99.49%
  • SEED-IV Dataset (4-class classification):
    • Single validation accuracy: 99.85%
    • 10-fold cross-validation accuracy: 99.70%

These results significantly outperform several existing deep learning and traditional machine learning methods, including CNN-GRU, CNN-LSTM, EEGNet, and VGG16-LSTM, as detailed in Tables 2, 3, 4, and 5. The model also showed strong performance in precision, recall, F1-score, and MCC across both datasets.

Limitations and Future Work

  • Computational Efficiency: The authors acknowledge that the model, while effective, is not a transformer and thus might require a large amount of data for training. They plan to optimize computational efficiency in future work by exploring more efficient network architectures (e.g., depthwise separable convolutions) and advanced training techniques.
  • Multimodal Fusion: To further enhance accuracy, future research could explore fusing MSBiLSTM-Attention with other physiological signals like electromyography (EMG) and eye movement data.
  • Subject Variability: The paper notes that significant differences in subject data within the SEED dataset can lead to fluctuations in experimental accuracy between individuals. Reducing this feature distribution disparity between subjects is identified as a key area for future exploration.

Summary updated

Abstract

Emotional states play a crucial role in shaping decision-making and social interactions, with sentiment analysis becoming an essential technology in human-computer emotional engagement, garnering increasing interest in artificial intelligence research. In EEG-based emotion analysis, the main challenges are feature extraction and classifier design, making the extraction of spatiotemporal information from EEG signals vital for effective emotion classification. Current methods largely depend on machine learning with manual feature extraction, while deep learning offers the advantage of automatic feature extraction and classification. Nonetheless, many deep learning approaches still necessitate manual preprocessing, which hampers accuracy and convenience. This paper introduces a novel deep learning technique that integrates multi-scale convolution and bidirectional long short-term memory networks with an attention mechanism for automatic EEG feature extraction and classification. By using raw EEG data, the method applies multi-scale convolutional neural networks and bidirectional long short-term memory networks to extract and merge features, selects key features via an attention mechanism, and classifies emotional EEG signals through a fully connected layer. The proposed model was evaluated on the SEED dataset for emotion classification. Experimental results demonstrate that this method effectively classifies EEG-based emotions, achieving classification accuracies of 99.44% for the three-class task and 99.85% for the four-class task in single validation, with average 10-fold-cross-validation accuracies of 99.49% and 99.70%, respectively. These findings suggest that the MSBiLSTM-Attention model is a powerful approach for emotion recognition.
BibTeX
@inproceedings{ma2025msbilstm,
  author = {Yahong Ma and Zhentao Huang and Yuyao Yang and Zuowen Chen and Qi Dong and Shanwen Zhang and Yuan-Fang Li},
  title = {{MSBiLSTM-Attention: EEG Emotion Recognition Model Based on Spatiotemporal Feature Fusion}},
  booktitle = {Biomimetics},
  year = {2025},
  doi = {10.3390/biomimetics10030178},
}
Figure