Learning Unseen Emotions from Gestures via Semantically-Conditioned Zero-Shot Perception with Adversarial Autoencoders

Abhishek Banerjee, Uttaran Bhattacharya, Aniket Bera

Abstract

Recognizing human emotion from body language is a fundamental challenge in affective computing. While supervised models excel at identifying common emotions like "happiness" or "sadness," they often fail to generalize to the full spectrum of human expression. This paper introduces a novel Generalized Zero-Shot Learning (GZSL) framework that identifies unseen emotion categories from 3D gestures by aligning motion-captured data with natural language embeddings. By leveraging a Semantically-Conditioned Adversarial Autoencoder (SC-AAE), the system achieves a 58.43% accuracy on unseen emotions, significantly outperforming existing state-of-the-art methods.

Introduction: The Challenge of the "Emotion Space"

Human emotion is nuanced and vast. Building labeled datasets for every possible affective state—such as "pride," "shame," or "relief"—is prohibitively expensive. Traditional machine learning models suffer from "seen-class bias," limiting their utility in real-world scenarios where they encounter emotions outside their training set. This research addresses the problem of Generalized Zero-Shot Learning (GZSL) for gestures, aiming to map 3D skeletal motion to emotion labels the model has never explicitly seen.

Methodology: Bridging Motion and Semantics

The proposed architecture consists of a two-stage pipeline designed to extract robust features and map them to a semantic space.

1. Feature Extraction (FS-GER)

The Fully Supervised Gesture Emotion Recognition (FS-GER) module utilizes Spatial-Temporal Graph Convolutional Networks (ST-GCN) to process 3D skeleton sequences. To enhance performance, the authors append "affective features"—hand-crafted metrics such as joint angles, movement jerkiness, and bounding box volume—derived from psychological principles.

2. Semantic Alignment (SC-AAE)

The core of the zero-shot capability lies in the Semantically-Conditioned Adversarial Autoencoder (SC-AAE). This module projects gesture features into a latent space aligned with word2vec embeddings of emotion labels.

The SC-AAE framework aligns gesture sequences with semantic embeddings to recognize unseen emotion classes like sadness.

Through adversarial training, the model learns to associate specific motion patterns with their linguistic counterparts. This allows the system to "infer" an unseen emotion (e.g., "Amusement") by identifying its proximity to seen emotions (e.g., "Joy") within the semantic vector space.

Experimental Results and Validation

The model was benchmarked using the MPI Emotional Body Expressions Database (EBEDB), which contains 1,447 3D motion-captured sequences across 11 emotion classes.

Key Performance Metrics

  • Supervised Accuracy: The FS-GER architecture achieved 77.61% accuracy on seen classes, a 7% improvement over previous benchmarks.
  • Zero-Shot Accuracy: In GZSL tasks, the model reached a harmonic mean accuracy of 58.43%, representing a 25–27% absolute improvement over standard zero-shot algorithms like CADA-VAE.

Qualitative "In-the-Wild" Validation

To test generalizability, the researchers applied the model to public video footage. By extracting 3D poses from 2D video frames, they demonstrated the model's ability to correctly classify emotions in unconstrained environments.

Qualitative results showing the model's ability to extract 3D poses from 2D video and correctly classify emotions in real-world scenes.

Insights and Future Directions

A critical finding of this study is the priority of the upper body in emotional expression. By focusing on 10 specific joints (head, torso, and arms) and ignoring the legs, the researchers reduced noise and improved accuracy. Furthermore, the hybrid approach of combining deep learning with hand-crafted affective features proved superior to using raw pose data alone.

Despite these gains, limitations remain. The use of generic word2vec embeddings, trained on general text, may not capture the full psychological nuance of affect. Future work will explore multi-modal inputs—including speech and eye movement—and transition toward continuous emotion models (Valence-Arousal-Dominance) to further refine zero-shot mapping.

Summary updated

Abstract

We present a novel generalized zero-shot algorithm to recognize perceived emotions from gestures. Our task is to map gestures to novel emotion categories not encountered in training. We introduce an adversarial autoencoder-based representation learning that correlates 3D motion-captured gesture sequences with the vectorized representation of the natural-language perceived emotion terms using word2vec embeddings. The language-semantic embedding provides a representation of the emotion label space, and we leverage this underlying distribution to map the gesture sequences to the appropriate categorical emotion labels. We train our method using a combination of gestures annotated with known emotion terms and gestures not annotated with any emotions. We evaluate our method on the MPI Emotional Body Expressions Database (EBEDB) and obtain an accuracy of 58.43%. We see an improvement in performance compared to current state-of-the-art algorithms for generalized zero-shot learning by an absolute 25-27%. We also demonstrate our approach on publicly available online videos and movie scenes, where the actors' pose has been extracted and map to their respective emotive states.
BibTeX
@inproceedings{banerjee2022learning,
  author = {Abhishek Banerjee and Uttaran Bhattacharya and Aniket Bera},
  title = {{Learning Unseen Emotions from Gestures via Semantically-Conditioned Zero-Shot Perception with Adversarial Autoencoders}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i1.19873},
}
Figure