Abstract
Emotion analysis is a crucial component in developing machines with true artificial intelligence. Human affective information is primarily tracked through electroencephalogram (EEG) signals and facial video sequences. Both signal types share common characteristics as spatial-temporal volumes. This paper proposes a novel deep learning framework, the Spatial-Temporal Recurrent Neural Network (STRNN), to unify the learning of these disparate signal sources into a single spatial-temporal dependency model. The STRNN employs a multi-directional recurrent neural network (RNN) layer to capture long-range contextual cues across spatial regions, followed by a bi-directional temporal RNN layer to learn discriminative temporal dependencies. Sparse projections are applied to hidden states to identify salient regions of emotion representation, enhancing model discriminability. Experimental results on public EEG and facial expression datasets demonstrate that STRNN outperforms state-of-the-art methods.
Introduction
Emotion recognition remains a complex challenge in human-machine interaction. Researchers typically rely on two primary signals: Electroencephalograms (EEG) and facial video sequences. However, emotions are not isolated events; they are patterns distributed across both space (different brain regions or facial muscles) and time.
Existing models often fail to integrate these dimensions effectively. Many treat EEG electrodes or facial pixels as independent features, ignoring spatial dependencies, or they focus exclusively on either spatial features (e.g., CNNs) or temporal sequences (e.g., standard RNNs). The STRNN architecture bridges this gap by modeling the inherent spatial-temporal volume of emotional data in a unified, end-to-end recurrent architecture.
The STRNN Architecture
The STRNN framework is designed to capture the "where" (spatial layout) and the "when" (temporal evolution) of human affect through a hierarchical structure.
1. Spatial RNN (SRNN) Layer
To capture spatial context, the model utilizes a quad-directional RNN. For any given time slice—whether a video frame or an EEG snapshot—the network scans the 2D grid from four different corners. This multi-directional traversal allows the model to capture long-range contextual cues, ensuring that every "pixel" or "electrode" can act as context for every other, creating a more robust representation than the local receptive fields of standard CNNs.
2. Sparse Projection and Saliency
Not all spatial regions contribute equally to emotion detection. The authors apply a sparse projection (using $l_1$-norm regularization) to the hidden states of the SRNN. This mechanism acts as an automated feature selector, "bundling" salient regions—such as the mouth and eyes in facial expressions—while suppressing noise from irrelevant areas. This global consideration significantly increases the model's discriminant ability.
3. Temporal RNN (TRNN) Layer
The spatial features produced by the SRNN are concatenated and fed into a bi-directional temporal RNN. This layer scans the sequence forward and backward to capture the long-term dynamics of how an emotion builds, peaks, and fades over time.
Experimental Evaluation
The model was evaluated on two distinct public datasets:
- SEED (SJTU Emotion EEG Dataset): 62-channel EEG signals from 15 subjects categorized into positive, neutral, and negative emotions.
- CK+ (Extended Cohn-Kanade): A facial expression dataset with 327 image sequences across seven emotion labels (anger, contempt, disgust, fear, happiness, sadness, and surprise).
Key Results
STRNN demonstrated superior performance over established baselines:
- EEG (SEED): Achieved an average accuracy of 89.50%, outperforming Deep Belief Networks (86.08%) and SVMs (83.99%).
- Facial Expression (CK+): Achieved an accuracy of 95.4%, surpassing 3D-CNNs (85.9%) and manifold-based methods (94.2%).
- Ablation Studies: Removing either the SRNN or the TRNN resulted in a performance drop (to approx. 85% on SEED), confirming that the integration of both dimensions is critical for success.
Insights and Future Work
The study revealed that spatiality is vital in EEG analysis; treating electrodes as a 2D grid rather than a flattened vector preserves essential discriminative information. Furthermore, the model confirmed neurological findings that high-frequency brain activity (Beta and Gamma bands) is most reliable for tracking emotional processing.
Despite its success, the model faces challenges in distinguishing "neutral" from low-intensity "negative" states in EEG data. Future research may involve refining the spatial layout by moving from 2D grids to complex 3D brain models to further enhance recognition accuracy.
Summary updated