MUGL: Large Scale Multi Person Conditional Action Generation With Locomotion

Shubh Maheshwari, Debtanu Gupta, Ravi Kiran SarvadevabhatlaWACV 2022

TL;DR

This paper introduces MUGL, a novel deep learning model for generating realistic, large-scale, and diverse human action sequences, including those involving locomotion and multiple people. MUGL addresses limitations in existing methods by enabling controllable, variable-length generations across over 100 action categories, paving the way for more practical applications in animation and robotics.

What Problem Does This Paper Address?

Generating realistic human motion is a complex task with significant applications in animation, robotics, and human-computer interaction. Previous approaches often struggled with:

  • Limited Scope: Many models could only generate a small number of action classes or were restricted to single-person actions.
  • Lack of Locomotion: Actions involving movement across a scene (like walking or running) were poorly handled.
  • Fixed Durations: Generated sequences often had a fixed length, making actions like "throwing" or "saluting" appear unnatural.
  • Data Quality Issues: Datasets like NTU-RGBD, while large, contained noisy and temporally incoherent 3D joint data from Kinect sensors, leading to suboptimal model training.
  • Difficulty in Multi-Person Scenarios: Synthesizing actions involving multiple interacting individuals with relative global movements was a significant challenge.

Methods

MUGL tackles these challenges with a novel deep neural network architecture and a hybrid pose representation:

  • Hybrid Pose Representation: The authors combine 3D pose sequences estimated from RGB videos (using VIBE, which provides more reliable local poses) with global trajectory information from Kinect-based sequences. This hybrid approach leverages the strengths of both data sources.
  • Decoupled Global and Local Components: To handle locomotion and relative movements, MUGL explicitly models the action sequence into two parts:
  • Global Component: Captures the trajectory of the root joint of each person, considering relative displacements between individuals.
  • Local Component: Represents the kinematic tree of each person's pose, with the root joint fixed at the origin. This component uses a continuous 6-D rotation representation for joint rotations, which is more robust than traditional methods.
  • Conditional Gaussian Mixture VAE (C-GMVAE): MUGL employs a C-GMVAE as its latent generative framework. This allows for modeling a richer, multi-modal latent space, enabling greater diversity in generated actions. Conditioning on action class and viewpoint further enhances control.
  • Encoder-Decoder Architecture: The model uses specialized encoder modules for the local pose, global trajectory, and sequence length, and corresponding decoder modules to reconstruct the action sequence.
  • Variable Sequence Length Handling: A dedicated sequence length encoder-decoder module allows MUGL to generate actions of varying durations, making the output more natural.

Datasets

  • NTU-RGBD-120: This large-scale dataset, containing 120 diverse single and multi-person activities, is used for training and evaluation. The authors leverage both the original Kinect-based 3D pose data and pose sequences extracted from RGB videos using VIBE.

Key Insights and Ideas

  • Hybrid Pose Representation is Crucial: Combining video-derived local poses with Kinect-derived global trajectories effectively addresses data quality issues and enables realistic motion generation, especially for locomotion.
  • Decoupling Global and Local Motion is Key for Locomotion: Explicitly modeling global trajectories and local pose dynamics separately allows the model to generate actions that involve movement through space and interaction between individuals.
  • 6-D Rotation Representation for Joints: Using a continuous 6-D representation for joint rotations bypasses issues with discontinuous representations and simplifies the optimization process.
  • C-GMVAE for Diversity: The Gaussian Mixture VAE with conditioning is essential for capturing the inherent diversity in human actions and generating varied outputs for the same action class.
  • Variable Length Generation is Necessary for Realism: Generating sequences of appropriate length, rather than fixed-length ones, significantly improves the naturalness of synthesized actions.

Key Results

  • Superior Generation Quality: MUGL outperforms strong baselines (VAE-LSTM, CS-GCN, SA-GCN, action2motion) in terms of generative quality scores (MMD-A and MMD-S) on the NTU-VIBE dataset.
  • Broad Category Coverage: MUGL successfully generates diverse actions across all 120 NTU-RGBD classes, including those with significant limb movement and locomotion.
  • Efficient Inference: MUGL achieves very fast inference times (0.02 seconds per sequence) compared to some baselines.
  • Qualitative Improvements: Visualizations show that MUGL generates more realistic and natural-looking sequences, especially for complex actions.
  • Ablation Studies Confirm Design Choices: Experiments show that the C-GMVAE, the 6-D rotation representation, and the variable sequence length mechanism all contribute significantly to MUGL's performance.

Limitations and Future Work

  • Limitations:
    • The paper notes that while MUGL's multi-person scores are comparable to its single-person variant, the configurations are different, making direct comparison difficult.
  • Feature-based generative quality measures (like FID, MMD) can be unreliable due to preprocessing steps that distort action dynamics.
  • Future Work:
    • The authors intend to explore MUGL's performance on other human pose action datasets.
  • The generated sequences can be used for rendering by mapping skeleton poses to SMPL system parameters.

Summary updated

BibTeX
@inproceedings{maheshwari2022mugl,
  author = {Shubh Maheshwari and Debtanu Gupta and Ravi Kiran Sarvadevabhatla},
  title = {{MUGL: Large Scale Multi Person Conditional Action Generation With Locomotion}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure