Spatial-Temporal Recurrent Neural Network for Emotion Recognition

Tong Zhang, Wenming Zheng, Zhen Cui, Yuan Zong, Yang Li

Abstract

Emotion analysis is a crucial component in developing machines with true artificial intelligence. Human affective information is primarily tracked through electroencephalogram (EEG) signals and facial video sequences. Both signal types share common characteristics as spatial-temporal volumes. This paper proposes a novel deep learning framework, the Spatial-Temporal Recurrent Neural Network (STRNN), to unify the learning of these disparate signal sources into a single spatial-temporal dependency model. The STRNN employs a multi-directional recurrent neural network (RNN) layer to capture long-range contextual cues across spatial regions, followed by a bi-directional temporal RNN layer to learn discriminative temporal dependencies. Sparse projections are applied to hidden states to identify salient regions of emotion representation, enhancing model discriminability. Experimental results on public EEG and facial expression datasets demonstrate that STRNN outperforms state-of-the-art methods.

Introduction

Emotion recognition remains a complex challenge in human-machine interaction. Researchers typically rely on two primary signals: Electroencephalograms (EEG) and facial video sequences. However, emotions are not isolated events; they are patterns distributed across both space (different brain regions or facial muscles) and time.

Existing models often fail to integrate these dimensions effectively. Many treat EEG electrodes or facial pixels as independent features, ignoring spatial dependencies, or they focus exclusively on either spatial features (e.g., CNNs) or temporal sequences (e.g., standard RNNs). The STRNN architecture bridges this gap by modeling the inherent spatial-temporal volume of emotional data in a unified, end-to-end recurrent architecture.

The STRNN Architecture

The STRNN framework is designed to capture the "where" (spatial layout) and the "when" (temporal evolution) of human affect through a hierarchical structure.

The overall architecture of the STRNN, illustrating the flow from spatial RNN layers to temporal RNN layers and final classification.

1. Spatial RNN (SRNN) Layer

To capture spatial context, the model utilizes a quad-directional RNN. For any given time slice—whether a video frame or an EEG snapshot—the network scans the 2D grid from four different corners. This multi-directional traversal allows the model to capture long-range contextual cues, ensuring that every "pixel" or "electrode" can act as context for every other, creating a more robust representation than the local receptive fields of standard CNNs.

2. Sparse Projection and Saliency

Not all spatial regions contribute equally to emotion detection. The authors apply a sparse projection (using $l_1$-norm regularization) to the hidden states of the SRNN. This mechanism acts as an automated feature selector, "bundling" salient regions—such as the mouth and eyes in facial expressions—while suppressing noise from irrelevant areas. This global consideration significantly increases the model's discriminant ability.

Visualization of salient regions identified by the sparse projection layer, highlighting key facial features used for emotion recognition.

3. Temporal RNN (TRNN) Layer

The spatial features produced by the SRNN are concatenated and fed into a bi-directional temporal RNN. This layer scans the sequence forward and backward to capture the long-term dynamics of how an emotion builds, peaks, and fades over time.

Experimental Evaluation

The model was evaluated on two distinct public datasets:

  • SEED (SJTU Emotion EEG Dataset): 62-channel EEG signals from 15 subjects categorized into positive, neutral, and negative emotions.
  • CK+ (Extended Cohn-Kanade): A facial expression dataset with 327 image sequences across seven emotion labels (anger, contempt, disgust, fear, happiness, sadness, and surprise).

Key Results

STRNN demonstrated superior performance over established baselines:

  • EEG (SEED): Achieved an average accuracy of 89.50%, outperforming Deep Belief Networks (86.08%) and SVMs (83.99%).
  • Facial Expression (CK+): Achieved an accuracy of 95.4%, surpassing 3D-CNNs (85.9%) and manifold-based methods (94.2%).
  • Ablation Studies: Removing either the SRNN or the TRNN resulted in a performance drop (to approx. 85% on SEED), confirming that the integration of both dimensions is critical for success.

Insights and Future Work

The study revealed that spatiality is vital in EEG analysis; treating electrodes as a 2D grid rather than a flattened vector preserves essential discriminative information. Furthermore, the model confirmed neurological findings that high-frequency brain activity (Beta and Gamma bands) is most reliable for tracking emotional processing.

Despite its success, the model faces challenges in distinguishing "neutral" from low-intensity "negative" states in EEG data. Future research may involve refining the spatial layout by moving from 2D grids to complex 3D brain models to further enhance recognition accuracy.

Summary updated

Abstract

Emotion analysis is a crucial problem to endow artifact machines with real intelligence in many large potential applications. As external appearances of human emotions, electroencephalogram (EEG) signals and video face signals are widely used to track and analyze human's affective information. According to their common characteristics of spatial-temporal volumes, in this paper we propose a novel deep learning framework named spatial-temporal recurrent neural network (STRNN) to unify the learning of two different signal sources into a spatial-temporal dependency model. In STRNN, to capture those spatially cooccurrent variations of human emotions, a multi-directional recurrent neural network (RNN) layer is employed to capture longrange contextual cues by traversing the spatial region of each time slice from multiple angles. Then a bi-directional temporal RNN layer is further used to learn discriminative temporal dependencies from the sequences concatenating spatial features of each time slice produced from the spatial RNN layer. To further select those salient regions of emotion representation, we impose sparse projection onto those hidden states of spatial and temporal domains, which actually also increases the model discriminant ability because of this global consideration. Consequently, such a two-layer RNN model builds spatial dependencies as well as temporal dependencies of the input signals. Experimental results on the public emotion datasets of EEG and facial expression demonstrate the proposed STRNN method is more competitive over those state-of-the-art methods.
BibTeX
@article{zhang2017spatial,
  author = {Tong Zhang and Wenming Zheng and Zhen Cui and Yuan Zong and Yang Li},
  title = {{Spatial-Temporal Recurrent Neural Network for Emotion Recognition}},
  journal = {IEEE Transactions on Cybernetics},
  year = {2017},
  doi = {10.1109/tcyb.2017.2788081},
}
Figure