Introduction
While emotion recognition often focuses on facial expressions or vocal cues, human body language offers a rich, privacy-preserving alternative. Facial analysis can be hindered by environmental factors like poor lighting or physical barriers like masks, and physiological sensors are often intrusive. Skeleton-based Emotion Recognition (SER) leverages 3D joint coordinates to identify affective states from posture and gait. This survey provides a unified framework for SER, bridging the gap between static posture analysis and dynamic gait patterns.
Technical Taxonomy of SER
The field has evolved through several technical paradigms, moving from manual feature engineering to automated deep learning. The authors categorize these into four primary approaches as illustrated in the survey's structural overview:
- Traditional Approaches: Utilizing handcrafted features such as joint angles and velocity with classic machine learning classifiers (e.g., SVM).
- Feat2Net: Hybrid models using handcrafted features as inputs for deep learning architectures like LSTMs.
- FeatFusionNet: Integrating both handcrafted features and raw skeletal data to leverage domain knowledge alongside automated extraction.
- End2EndNet: State-of-the-art models, such as Graph Convolutional Networks (GCNs) and Transformers, that learn representations directly from raw sequences.
Key Insights: The Arousal Advantage and Model Architectures
A significant finding in SER is the "Arousal Advantage." Body movement is particularly effective at conveying the intensity of an emotion (arousal) rather than its positivity or negativity (valence). For instance, high-energy emotions like "angry" and "happy" are easier to distinguish from low-energy "sadness" than from each other. This is often mapped using standard psychological models like the Valence-Arousal (VA) space or the Pleasure-Arousal-Dominance (PAD) model.
Furthermore, Graph Convolutional Networks (GCNs) have emerged as the dominant architecture. By treating the human skeleton as a natural graph where joints are nodes and bones are edges, GCNs effectively capture the spatial relationships inherent in human anatomy. Interestingly, handcrafted features remain relevant in data-scarce scenarios, as they incorporate established psychological markers like head tilt.
Benchmarking and Datasets
The survey reviews key benchmarks, highlighting a transition from laboratory Motion Capture (MoCap) data to "in-the-wild" vision-based systems.
- Posture-based Datasets: Emilya and BML remain standard for high-precision samples recorded with professional suits.
- Gait-based Datasets: E-Gait and EMOGAIT focus on real-world walking patterns extracted from video.
- Performance: On the EGBM dataset, manual features with fully connected networks reached 97.43% accuracy. In gait analysis, the Multi-Scale Adaptive GCN (MSA-GCN) leads with 93.51% accuracy on E-Gait, demonstrating the power of modeling movement at different time scales.
Future Research Directions
Despite high accuracy in controlled environments, several challenges remain for real-world application:
- Diversity: There is a need for datasets encompassing broader age groups and cultures beyond acted environments.
- Group Affect: Moving from single-person analysis to multi-person social interactions and "group affect."
- Multimodal Fusion: Combining skeletal data with voice and facial cues for more robust recognition.
- Efficiency: Optimizing models for real-time deployment on edge devices like home robots or wearable health monitors.
Summary updated