Action-conditioned On-demand Motion Generation

Qiujing Lu, Yipeng Zhang, Mingjian Lu, Vwani Roychowdhury

TL;DR

This paper introduces ODMO, a novel framework for generating realistic and diverse 3D human motion sequences conditioned solely on action type. ODMO significantly outperforms existing methods and introduces unprecedented customization capabilities, including discovering and interpolating motion styles, and precisely controlling trajectory endpoints.

What Problem Does This Paper Address?

Generating realistic and controllable 3D human motion sequences on demand is a long-standing goal in multimedia and animation. Existing methods often require initial motion frames or intermediate poses as input, limiting their "on-demand" applicability. Furthermore, while some models can generate motions based on action types, they lack fine-grained control over motion styles within an action or the precise trajectory of the motion. This paper aims to bridge this gap by enabling action-conditioned motion generation with built-in capabilities for discovering diverse motion subtypes, interpolating between them, and customizing the motion's trajectory to a specified endpoint.

Methods

ODMO employs a novel encoder-decoder architecture with several key innovations:

  • Hierarchical Latent Space with Contrastive Learning (Encoder): The encoder learns a low-dimensional representation of motion sequences. Crucially, contrastive learning is used to structure this latent space hierarchically. This means that motions of different action types are separated into distinct clusters, and within each action type, similar motion styles (modes) are further grouped together. This makes these modes discoverable.
  • Hierarchical Decoding with Trajectory Control (Decoder): The decoder first reconstructs the motion's trajectory (the path of the body's center) and then uses this trajectory, along with the latent code, to generate the full motion sequence. This hierarchical approach allows for precise control over the motion's trajectory.
  • Mode Discovery and Multi-modal Sampling: By fitting Gaussian Mixture Models (GMMs) to the latent codes within each action category, ODMO can automatically discover different motion styles (modes). These modes can then be sampled to generate diverse motions, or interpolated to create novel motion sequences.

Datasets

The authors evaluated ODMO on three public 3D human motion datasets:

  • HumanAct12: Contains 1,191 motion clips across 12 action categories.
  • UESTC: A larger dataset with 40 action types performed by 118 subjects, totaling 25,600 video samples.
  • CMU MoCap (MoCap): Features 970 motion sequences across 7 action categories, noted for its longer overall spatial translations and greater diversity.

Key Insights and Ideas

  • Latent Space Structure is Key: The paper strongly emphasizes that the way the latent space is organized is critical. By using contrastive learning, they achieve a latent space that not only separates action types but also reveals distinct motion "styles" or "modes" within each type. This is a significant departure from simply clustering by action.
  • Hierarchical Decoding Enables Control: The two-stage decoding process (trajectory first, then full motion) is a clever way to decouple trajectory generation from the finer details of joint movements, enabling precise endpoint control.
  • Automated Discovery of Motion Styles: The ability to automatically discover and sample from different motion modes within an action category (e.g., different ways of "walking") is a powerful feature that opens up new avenues for animation and data augmentation.
  • Customization is More Than Just Style: Beyond style transfer, the ability to specify the destination of the motion is a crucial step towards truly programmable motion generation.

Key Results

  • State-of-the-Art Performance: ODMO achieved significant improvements over existing state-of-the-art (SOTA) methods (Action2Motion, ACTOR) across various metrics, including Recognition Accuracy (Acc), Frechet Inception Distance (FID), Diversity, and Multimodality, on all three datasets. For example, on MoCap, ODMO achieved a 93.51% Acc, outperforming ACTOR (77.49%) by a substantial margin.
  • Unprecedented Customization:
    • Mode Discovery: ODMO successfully identified distinct modes within action categories like "Warm Up" and "dumbbell-one-arm-shoulder-pressing," visualized using t-SNE projections of latent codes.
  • Motion Interpolation: The paper demonstrated smooth interpolations between discovered motion modes, creating novel motion styles.
  • Trajectory Customization: ODMO could generate realistic motions that reached specified end locations with high accuracy (low diste metric), showcasing its ability to control the motion's path.

Limitations and Future Work

  • Dataset Dependence: While ODMO performs well across datasets, the paper notes that MoCap's complexity (longer spatial translations) presents a greater challenge, and A2M struggles more on this dataset compared to ODMO.
  • Computational Cost: Training complex generative models like ODMO is computationally intensive, though the paper does not explicitly detail the training times.
  • Future Work: The authors suggest that their systematic data generation capabilities can be used for data augmentation in style transfer tasks. The framework's ability to generate diverse and controllable motions also widens the spectrum of potential applications for 3D human motion generators.

Summary updated

BibTeX
@inproceedings{lu2022action,
  author = {Qiujing Lu and Yipeng Zhang and Mingjian Lu and Vwani Roychowdhury},
  title = {{Action-conditioned On-demand Motion Generation}},
  booktitle = {ACM Multimedia},
  year = {2022},
  url = {https://openreview.net/forum?id=W2hdO2rgrE-},
}
Figure