Emotion Transfer for 3D Hand and Full Body Motion Using StarGAN

Jacky C. P. Chan, Edmond S. L. HoComput 2021 · DOI · OpenReview

TL;DR

This paper introduces a novel framework for transferring emotions to 3D hand and full-body motion sequences by treating motion synthesis as an image-to-image translation problem using StarGAN. The method effectively synthesizes new motions with target emotions, demonstrating comparable or improved visual quality and naturalness to captured data and existing methods.

What Problem Does This Paper Address?

Animating 3D characters with realistic emotional expressions is crucial for immersion, but while facial expressions have been extensively studied, expressing emotions through body language, particularly hand gestures, has received less attention. Synthesizing expressive hand and full-body motions that convey specific emotions is challenging due to the complexity of capturing and representing fine-grained movements. Existing methods often struggle with transferring styles or require extensive manual control. This paper aims to bridge this gap by providing a data-driven framework that automatically transfers emotional styles to existing motion sequences.

Methods

The core of the proposed method is to frame motion synthesis as an image-to-image translation task, leveraging the power of Generative Adversarial Networks (GANs), specifically StarGAN.

  1. Motion to Image Representation: 3D motion data (joint angles/positions and velocities) is converted into an image format. Each keyframe of a motion becomes a row in the image, and each joint's spatial or angular information is mapped to the RGB channels. This transformation allows the use of 2D CNN-based architectures.
  2. Emotion Transfer with StarGAN: The StarGAN model, known for its ability to handle multi-domain image-to-image translation, is employed. It learns to translate an input motion image from one emotion domain to another, preserving the underlying motion structure while altering the emotional style.
  3. Image to Motion Reconstruction: The synthesized emotion-conditioned image is then converted back into a 3D motion sequence by reconstructing joint angles/positions and interpolating between keyframes.

The use of StarGAN is highlighted as an advantage for its ability to handle multiple emotion domains within a single model, making it scalable and robust.

Datasets

The study utilizes two primary datasets:

  • Hand Motion Dataset: This dataset, collected in the authors' previous work, contains 35 hand motions across seven action types, each captured with five different emotions (Angry, Happy, Neutral, Sad, Fearful).
  • Full Body Motion Dataset: This dataset is sourced from the publicly available Body Movement Library, containing 4080 full-body motions from 30 participants. These motions cover three action types (knocking, lifting, throwing) and four emotion states (Neutral, Angry, Happy, Sad).

The paper emphasizes the standardization of motion data as a crucial preprocessing step.

Key Insights and Ideas

  • Motion as Image: The novel idea of representing 3D motion as a 2D image for style transfer is a key contribution. This allows leveraging powerful image-based deep learning models like StarGAN.
  • StarGAN for Multi-Emotion Transfer: Applying StarGAN, originally designed for image domains, to the multi-domain emotion transfer problem in motion synthesis is a significant adaptation. It enables efficient transfer across various emotions without needing separate models for each pair.
  • Hand Motion Significance: The paper reiterates the importance of hand gestures in conveying emotion and personality, drawing parallels to characters like Thing T. Thing from "The Addams Family."
  • Emotion vs. Style: While previous work focused on interpolating emotion strength, this paper focuses on transferring distinct emotional styles, offering a richer expressive capability.

Key Results

  • User Study (Hand Motions):
    • A user study showed an average emotion recognition accuracy of 65.83% for both captured and synthesized hand motions, indicating that the synthesized motions convey emotions effectively.
  • Qualitative comparisons showed that synthesized motions were rated as "more pleasant" or equally pleasant as captured motions, with statistical significance suggesting the method does not degrade visual quality.
  • Comparison with a PCA-based method (Irimia et al. [15]) showed the proposed method achieved better visual quality, with a statistically significant improvement when considering "do not know" responses.
  • Qualitative Evaluation (Hand & Full Body):
    • Synthesized hand motions visually exhibited characteristics of the target emotions (e.g., exaggerated movements for anger, slower movements for sadness, asynchronous fingers for fear).
  • Full-body motion synthesis also demonstrated consistent emotional trends in speed and movement style across different emotions.

Limitations and Future Work

  • Personal Style: The authors acknowledge that their framework primarily focuses on emotion transfer and does not explicitly model individual "personal style" differences in motion. This can lead to slight discrepancies between synthesized and captured motions.
  • Future Work: Incorporating personal style into the motion modeling framework is a key future direction. This could involve disentangling "base" motion, personal style, and emotion.
  • Multi-Emotion Labels: The user study revealed ambiguity in emotion perception, suggesting that a single emotion label might be insufficient.
  • Future Work: Exploring the feasibility of having multiple emotion labels with varying strengths for motion representation and synthesis is suggested.
  • Quantitative Evaluation: While user studies provide qualitative insights, more quantitative metrics for evaluating motion differences are needed.
  • Future Work: Developing methods to quantitatively evaluate the differences between synthesized and ground-truth motions is proposed.
  • Data Capture Challenges: Capturing high-quality global translation and rotation for hand motions remains a challenge.
  • Future Work: Utilizing state-of-the-art MOCAP solutions is suggested to overcome these capture limitations.
  • Image Representation: The current image representation using 2D CNNs might not optimally capture the tree-like skeletal structure of motions.
  • Future Work: Exploring alternative representations like Graph Convolutional Networks (GCNs) is suggested for better modeling of skeletal structures.

Summary updated

Abstract

In this paper, we propose a new data-driven framework for 3D hand and full-body motion emotion transfer. Specifically, we formulate the motion synthesis task as an image-to-image translation problem. By presenting a motion sequence as an image representation, the emotion can be transferred by our framework using StarGAN. To evaluate our proposed method’s effectiveness, we first conducted a user study to validate the perceived emotion from the captured and synthesized hand motions. We further evaluate the synthesized hand and full body motions qualitatively and quantitatively. Experimental results show that our synthesized motions are comparable to the captured motions and those created by an existing method in terms of naturalness and visual quality.
BibTeX
@article{chan2021emotion,
  author = {Jacky C. P. Chan and Edmond S. L. Ho},
  title = {{Emotion Transfer for 3D Hand and Full Body Motion Using StarGAN}},
  journal = {comput.},
  year = {2021},
  doi = {10.3390/computers10030038},
}
Figure