Emotion Recognition from Skeleton Data: A Comprehensive Survey

Haifeng Lu, Jiuyi Chen, Zhen Zhang, Ruida Liu, Runhao Zeng, Xiping HuarXiv 2025 · DOI · arXiv

Introduction

While emotion recognition often focuses on facial expressions or vocal cues, human body language offers a rich, privacy-preserving alternative. Facial analysis can be hindered by environmental factors like poor lighting or physical barriers like masks, and physiological sensors are often intrusive. Skeleton-based Emotion Recognition (SER) leverages 3D joint coordinates to identify affective states from posture and gait. This survey provides a unified framework for SER, bridging the gap between static posture analysis and dynamic gait patterns.

Technical Taxonomy of SER

The field has evolved through several technical paradigms, moving from manual feature engineering to automated deep learning. The authors categorize these into four primary approaches as illustrated in the survey's structural overview:

  1. Traditional Approaches: Utilizing handcrafted features such as joint angles and velocity with classic machine learning classifiers (e.g., SVM).
  2. Feat2Net: Hybrid models using handcrafted features as inputs for deep learning architectures like LSTMs.
  3. FeatFusionNet: Integrating both handcrafted features and raw skeletal data to leverage domain knowledge alongside automated extraction.
  4. End2EndNet: State-of-the-art models, such as Graph Convolutional Networks (GCNs) and Transformers, that learn representations directly from raw sequences.
A hierarchical taxonomy of the survey covering models, data, recognition methods, and applications.

Key Insights: The Arousal Advantage and Model Architectures

A significant finding in SER is the "Arousal Advantage." Body movement is particularly effective at conveying the intensity of an emotion (arousal) rather than its positivity or negativity (valence). For instance, high-energy emotions like "angry" and "happy" are easier to distinguish from low-energy "sadness" than from each other. This is often mapped using standard psychological models like the Valence-Arousal (VA) space or the Pleasure-Arousal-Dominance (PAD) model.

Commonly used emotion models: (a) 2D Valence-Arousal (VA), (b) 3D Pleasure-Arousal-Dominance (PAD), and (c) Plutchik’s wheel of emotions.

Furthermore, Graph Convolutional Networks (GCNs) have emerged as the dominant architecture. By treating the human skeleton as a natural graph where joints are nodes and bones are edges, GCNs effectively capture the spatial relationships inherent in human anatomy. Interestingly, handcrafted features remain relevant in data-scarce scenarios, as they incorporate established psychological markers like head tilt.

Benchmarking and Datasets

The survey reviews key benchmarks, highlighting a transition from laboratory Motion Capture (MoCap) data to "in-the-wild" vision-based systems.

  • Posture-based Datasets: Emilya and BML remain standard for high-precision samples recorded with professional suits.
  • Gait-based Datasets: E-Gait and EMOGAIT focus on real-world walking patterns extracted from video.
  • Performance: On the EGBM dataset, manual features with fully connected networks reached 97.43% accuracy. In gait analysis, the Multi-Scale Adaptive GCN (MSA-GCN) leads with 93.51% accuracy on E-Gait, demonstrating the power of modeling movement at different time scales.

Future Research Directions

Despite high accuracy in controlled environments, several challenges remain for real-world application:

  • Diversity: There is a need for datasets encompassing broader age groups and cultures beyond acted environments.
  • Group Affect: Moving from single-person analysis to multi-person social interactions and "group affect."
  • Multimodal Fusion: Combining skeletal data with voice and facial cues for more robust recognition.
  • Efficiency: Optimizing models for real-time deployment on edge devices like home robots or wearable health monitors.

Summary updated

Abstract

Emotion recognition through body movements has emerged as a compelling and privacy-preserving alternative to traditional methods that rely on facial expressions or physiological signals. Recent advancements in 3D skeleton acquisition technologies and pose estimation algorithms have significantly enhanced the feasibility of emotion recognition based on full-body motion. This survey provides a comprehensive and systematic review of skeleton-based emotion recognition techniques. First, we introduce psychological models of emotion and examine the relationship between bodily movements and emotional expression. Next, we summarize publicly available datasets, highlighting the differences in data acquisition methods and emotion labeling strategies. We then categorize existing methods into posture-based and gait-based approaches, analyzing them from both data-driven and technical perspectives. In particular, we propose a unified taxonomy that encompasses four primary technical paradigms: Traditional approaches, Feat2Net, FeatFusionNet, and End2EndNet. Representative works within each category are reviewed and compared, with benchmarking results across commonly used datasets. Finally, we explore the extended applications of emotion recognition in mental health assessment, such as detecting depression and autism, and discuss the open challenges and future research directions in this rapidly evolving field.
BibTeX
@misc{lu2025emotion,
  author = {Haifeng Lu and Jiuyi Chen and Zhen Zhang and Ruida Liu and Runhao Zeng and Xiping Hu},
  title = {{Emotion Recognition from Skeleton Data: A Comprehensive Survey}},
  year = {2025},
  eprint = {2507.18026},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.18026},
}
Figure