Abstract
Recognizing human emotion from body language is a fundamental challenge in affective computing. While supervised models excel at identifying common emotions like "happiness" or "sadness," they often fail to generalize to the full spectrum of human expression. This paper introduces a novel Generalized Zero-Shot Learning (GZSL) framework that identifies unseen emotion categories from 3D gestures by aligning motion-captured data with natural language embeddings. By leveraging a Semantically-Conditioned Adversarial Autoencoder (SC-AAE), the system achieves a 58.43% accuracy on unseen emotions, significantly outperforming existing state-of-the-art methods.
Introduction: The Challenge of the "Emotion Space"
Human emotion is nuanced and vast. Building labeled datasets for every possible affective state—such as "pride," "shame," or "relief"—is prohibitively expensive. Traditional machine learning models suffer from "seen-class bias," limiting their utility in real-world scenarios where they encounter emotions outside their training set. This research addresses the problem of Generalized Zero-Shot Learning (GZSL) for gestures, aiming to map 3D skeletal motion to emotion labels the model has never explicitly seen.
Methodology: Bridging Motion and Semantics
The proposed architecture consists of a two-stage pipeline designed to extract robust features and map them to a semantic space.
1. Feature Extraction (FS-GER)
The Fully Supervised Gesture Emotion Recognition (FS-GER) module utilizes Spatial-Temporal Graph Convolutional Networks (ST-GCN) to process 3D skeleton sequences. To enhance performance, the authors append "affective features"—hand-crafted metrics such as joint angles, movement jerkiness, and bounding box volume—derived from psychological principles.
2. Semantic Alignment (SC-AAE)
The core of the zero-shot capability lies in the Semantically-Conditioned Adversarial Autoencoder (SC-AAE). This module projects gesture features into a latent space aligned with word2vec embeddings of emotion labels.
Through adversarial training, the model learns to associate specific motion patterns with their linguistic counterparts. This allows the system to "infer" an unseen emotion (e.g., "Amusement") by identifying its proximity to seen emotions (e.g., "Joy") within the semantic vector space.
Experimental Results and Validation
The model was benchmarked using the MPI Emotional Body Expressions Database (EBEDB), which contains 1,447 3D motion-captured sequences across 11 emotion classes.
Key Performance Metrics
- Supervised Accuracy: The FS-GER architecture achieved 77.61% accuracy on seen classes, a 7% improvement over previous benchmarks.
- Zero-Shot Accuracy: In GZSL tasks, the model reached a harmonic mean accuracy of 58.43%, representing a 25–27% absolute improvement over standard zero-shot algorithms like CADA-VAE.
Qualitative "In-the-Wild" Validation
To test generalizability, the researchers applied the model to public video footage. By extracting 3D poses from 2D video frames, they demonstrated the model's ability to correctly classify emotions in unconstrained environments.
Insights and Future Directions
A critical finding of this study is the priority of the upper body in emotional expression. By focusing on 10 specific joints (head, torso, and arms) and ignoring the legs, the researchers reduced noise and improved accuracy. Furthermore, the hybrid approach of combining deep learning with hand-crafted affective features proved superior to using raw pose data alone.
Despite these gains, limitations remain. The use of generic word2vec embeddings, trained on general text, may not capture the full psychological nuance of affect. Future work will explore multi-modal inputs—including speech and eye movement—and transition toward continuous emotion models (Valence-Arousal-Dominance) to further refine zero-shot mapping.
Summary updated