ASMNet: Action and Style-Conditioned Motion Generative Network for 3D Human Motion Generation

Zongying Li, Yong Wang, Xin Du, Can Wang, Reinhard Koch, Mengyuan LiuCyborg 2024 · DOI

TL;DR

This paper introduces ASMNet, a novel network for generating 3D human motion that not only performs a specified action but also imbues it with a distinct stylistic flair. By leveraging real motion clips as style examples and employing a specialized spatial-temporal extractor, ASMNet achieves more realistic and expressive motion generation than previous methods.

What Problem Does This Paper Address?

Generating realistic 3D human motion is a challenging task, and existing methods often fall short of capturing the nuances of human movement. While many approaches focus on generating motions based on action labels (e.g., "walking," "running"), they struggle to incorporate stylistic elements that convey emotion, personality, or intent. For instance, "walking" can look vastly different depending on whether someone is happy, sad, or angry. This is partly due to the difficulty of capturing motion data with diverse styles and the abstract nature of "style" itself, making it hard to describe with text alone. Current methods that attempt to incorporate style often produce generic results or lack clear stylistic differentiation. This paper aims to bridge this gap by enabling the generation of human motion that is both action-accurate and stylistically rich, using real motion examples to guide the style.

Methods

ASMNet is built upon a Conditional Variational Autoencoder (CVAE) framework, comprising two main branches: motion generation and style injection.

  • Motion Generation: This branch uses a novel spatial-temporal extractor within its motion encoder and decoder. This extractor is designed to capture both local spatial correlations between joints within a single frame and global temporal correlations across frames in a sequence. This is achieved using Transformer-based blocks. The motion encoder takes the motion sequence and action-specific learnable tokens as input, producing parameters for a latent distribution from which a latent vector ($z^M$) is sampled. This latent vector, along with the action label ($a$) and duration ($F$), is fed into the motion decoder to reconstruct the motion sequence.
  • Style Injection: This branch utilizes a style extractor to learn latent style features ($z^S$) from a provided source motion clip. These extracted style features are then injected into the generated motion sequence using Adaptive Instance Normalization (AdaIN) layers within a style injector. AdaIN is crucial here as it normalizes the generated motion's feature statistics and then re-scales them based on the learned style features, effectively transferring the style.

The model is trained using a combination of reconstruction loss (for both action generation and style injection), KL divergence loss (to regularize the latent space), and content preservation loss (to ensure style injection doesn't corrupt the original action).

Datasets

  • Xia dataset [18]: Used for training and evaluating the style transfer capabilities. It contains motion capture data with six actions and eight styles, totaling 11 minutes of motion.
  • HumanAct12 dataset [17]: Used for evaluating action generation performance. It includes 1,191 motion clips across 12 action categories.

Key Insights and Ideas

  • Real motion as style source: The paper argues that abstract motion styles are difficult to describe with text. Using actual motion clips as style examples provides a more direct and effective way to capture and transfer these nuances.
  • Spatial-temporal extractor: The novel extractor is key to capturing both fine-grained joint relationships within a frame and the temporal dynamics of motion, leading to more realistic and detailed motion.
  • AdaIN for style transfer: Adaptive Instance Normalization is a powerful technique for style transfer, allowing the model to effectively imbue generated motion with the characteristics of a reference style.
  • Unified framework: ASMNet combines action generation and style injection into a single, coherent framework, addressing both aspects simultaneously.

Key Results

  • Quantitative Performance: ASMNet achieves state-of-the-art or comparable results on both action generation (HumanAct12) and style transfer (Xia dataset) benchmarks. For instance, on HumanAct12, it shows strong FID scores, accuracy, and multimodality. On the Xia dataset, it demonstrates a significant improvement in Content Recognition Accuracy (CRA) for style transfer compared to previous methods.
  • Qualitative Performance: Visualizations show that ASMNet generates motions that are not only action-correct but also exhibit distinct and visually appealing stylistic variations (e.g., depressed, strutting, old, angry walking styles), outperforming methods like MotionCLIP and MotionDiffuse which tend to produce more generic results.

Limitations and Future Work

  • Data Scarcity: The authors acknowledge that current motion style research relies heavily on motion capture data, which can be scarce for diverse styles.
  • Future Work: They suggest exploring methods to obtain sufficient motion style data from other sources to enable the generation of motions with even more diverse styles.

Summary updated

Abstract

Extensive research has explored human motion generation, but the generated sequences are influenced by different motion styles. For instance, the act of walking with joy and sorrow evokes distinct effects on a character's motion. Due to the difficulties in motion capture with styles, the available data for style research are also limited. To address the problems, we propose ASMNet, an action and style-conditioned motion generative network. This network ensures that the generated human motion sequences not only comply with the provided action label but also exhibit distinctive stylistic features. To extract motion features from human motion sequences, we design a spatial temporal extractor. Moreover, we use the adaptive instance normalization layer to inject style into the target motion. Our results are comparable to state-of-the-art approaches and display a substantial advantage in both quantitative and qualitative evaluations. The code is available at https://github.com/ZongYingLi/ASMNet.git.
BibTeX
@inproceedings{li2024asmnet,
  author = {Zongying Li and Yong Wang and Xin Du and Can Wang and Reinhard Koch and Mengyuan Liu},
  title = {{ASMNet: Action and Style-Conditioned Motion Generative Network for 3D Human Motion Generation}},
  booktitle = {Cyborg},
  year = {2024},
  doi = {10.34133/cbsystems.0090},
}
Figure