TL;DR
This paper introduces MUGL, a novel deep learning model for generating realistic, large-scale, and diverse human action sequences, including those involving locomotion and multiple people. MUGL addresses limitations in existing methods by enabling controllable, variable-length generations across over 100 action categories, paving the way for more practical applications in animation and robotics.
What Problem Does This Paper Address?
Generating realistic human motion is a complex task with significant applications in animation, robotics, and human-computer interaction. Previous approaches often struggled with:
- Limited Scope: Many models could only generate a small number of action classes or were restricted to single-person actions.
- Lack of Locomotion: Actions involving movement across a scene (like walking or running) were poorly handled.
- Fixed Durations: Generated sequences often had a fixed length, making actions like "throwing" or "saluting" appear unnatural.
- Data Quality Issues: Datasets like NTU-RGBD, while large, contained noisy and temporally incoherent 3D joint data from Kinect sensors, leading to suboptimal model training.
- Difficulty in Multi-Person Scenarios: Synthesizing actions involving multiple interacting individuals with relative global movements was a significant challenge.
Methods
MUGL tackles these challenges with a novel deep neural network architecture and a hybrid pose representation:
- Hybrid Pose Representation: The authors combine 3D pose sequences estimated from RGB videos (using VIBE, which provides more reliable local poses) with global trajectory information from Kinect-based sequences. This hybrid approach leverages the strengths of both data sources.
- Decoupled Global and Local Components: To handle locomotion and relative movements, MUGL explicitly models the action sequence into two parts:
- Global Component: Captures the trajectory of the root joint of each person, considering relative displacements between individuals.
- Local Component: Represents the kinematic tree of each person's pose, with the root joint fixed at the origin. This component uses a continuous 6-D rotation representation for joint rotations, which is more robust than traditional methods.
- Conditional Gaussian Mixture VAE (C-GMVAE): MUGL employs a C-GMVAE as its latent generative framework. This allows for modeling a richer, multi-modal latent space, enabling greater diversity in generated actions. Conditioning on action class and viewpoint further enhances control.
- Encoder-Decoder Architecture: The model uses specialized encoder modules for the local pose, global trajectory, and sequence length, and corresponding decoder modules to reconstruct the action sequence.
- Variable Sequence Length Handling: A dedicated sequence length encoder-decoder module allows MUGL to generate actions of varying durations, making the output more natural.
Datasets
- NTU-RGBD-120: This large-scale dataset, containing 120 diverse single and multi-person activities, is used for training and evaluation. The authors leverage both the original Kinect-based 3D pose data and pose sequences extracted from RGB videos using VIBE.
Key Insights and Ideas
- Hybrid Pose Representation is Crucial: Combining video-derived local poses with Kinect-derived global trajectories effectively addresses data quality issues and enables realistic motion generation, especially for locomotion.
- Decoupling Global and Local Motion is Key for Locomotion: Explicitly modeling global trajectories and local pose dynamics separately allows the model to generate actions that involve movement through space and interaction between individuals.
- 6-D Rotation Representation for Joints: Using a continuous 6-D representation for joint rotations bypasses issues with discontinuous representations and simplifies the optimization process.
- C-GMVAE for Diversity: The Gaussian Mixture VAE with conditioning is essential for capturing the inherent diversity in human actions and generating varied outputs for the same action class.
- Variable Length Generation is Necessary for Realism: Generating sequences of appropriate length, rather than fixed-length ones, significantly improves the naturalness of synthesized actions.
Key Results
- Superior Generation Quality: MUGL outperforms strong baselines (VAE-LSTM, CS-GCN, SA-GCN, action2motion) in terms of generative quality scores (MMD-A and MMD-S) on the NTU-VIBE dataset.
- Broad Category Coverage: MUGL successfully generates diverse actions across all 120 NTU-RGBD classes, including those with significant limb movement and locomotion.
- Efficient Inference: MUGL achieves very fast inference times (0.02 seconds per sequence) compared to some baselines.
- Qualitative Improvements: Visualizations show that MUGL generates more realistic and natural-looking sequences, especially for complex actions.
- Ablation Studies Confirm Design Choices: Experiments show that the C-GMVAE, the 6-D rotation representation, and the variable sequence length mechanism all contribute significantly to MUGL's performance.
Limitations and Future Work
- Limitations:
- The paper notes that while MUGL's multi-person scores are comparable to its single-person variant, the configurations are different, making direct comparison difficult.
- Feature-based generative quality measures (like FID, MMD) can be unreliable due to preprocessing steps that distort action dynamics.
- Future Work:
- The authors intend to explore MUGL's performance on other human pose action datasets.
- The generated sequences can be used for rendering by mapping skeleton poses to SMPL system parameters.
Summary updated