DSAG: A Scalable Deep Framework for Action-Conditioned Multi-Actor Full Body Motion Synthesis

Debtanu Gupta, Shubh Maheshwari, Sai Shashank Kalakonda, Manasvi Vaidyula, Ravi Kiran SarvadevabhatlaWACV 2023

TL;DR

This paper introduces DSAG, a novel deep learning framework for generating realistic, full-body, multi-actor human motion sequences conditioned on specific actions. DSAG addresses limitations in existing methods by introducing dedicated representations for fine-grained finger joints and a scalable architecture that handles variable durations, actor counts, and diverse datasets.

What Problem Does This Paper Address?

Generating realistic 3D human motion from action labels is a challenging task. Existing methods often struggle with:

  • Limited Detail: Many large-scale datasets lack detailed finger joint information, leading to unrealistic hand movements.
  • Single-Actor Focus: Most approaches are designed for single-person actions, limiting their applicability to real-world scenarios with multiple interacting individuals.
  • Fixed Durations: Generating actions of variable lengths is difficult, as most models are trained on fixed-duration sequences.
  • Dataset Inconsistency: Models trained on one dataset often fail to generalize to others due to variations in joint counts, frame rates, and action categories.
  • Temporal Incoherence: Datasets derived from sources like Kinect can have poorly estimated joints and temporal inconsistencies, impacting the quality of generated motions.

DSAG aims to overcome these limitations by providing a scalable, controllable, and versatile framework for action-conditioned motion synthesis.

Methods

DSAG employs a conditional variational autoencoder (CVAE) architecture with several key innovations:

  • Decoupled Representations: The framework separates motion into local and global components.
  • Local Components: These capture the detailed pose of individual body parts. Crucially, DSAG introduces dedicated representations for local hand joints (fingers), addressing the lack of detail in many datasets. This is achieved by treating finger joints as a separate kinematic tree.
  • Global Components: These represent the overall body and hand trajectories (e.g., root joint movement for locomotion). For multi-actor scenarios, relative displacements are used to capture interactions.
  • Spatio-Temporal (ST) Blocks: These novel blocks form the core of the encoder and decoder. They combine:
  • Spatial Encoding: Compresses spatial information within each timestep.
  • Temporal Encoding: Uses local filters to capture subtle temporal movements and handle high within-class action variance.
  • Multi-Head Temporal Self-Attention (MHTS): Captures long-term global dependencies within the action sequence, crucial for realistic motion.
  • Variable Duration Handling: A dedicated sequence length decoder models the temporal duration of actions, allowing for variable-length outputs.
  • Action Conditioning: The model is conditioned on action class labels, guiding the generation process.

Datasets

DSAG is trained and evaluated on several large-scale 3D human motion datasets:

  • NTU-RGBD-120: A large dataset with 120 action categories, used for both single and multi-person scenarios. The authors used ExPose to extract 52-joint sequences with detailed finger joints.
  • NTU-VIBE: A coarser-grained version of NTU-RGBD, used for comparison.
  • HumanAct12: Contains 12 action categories, mostly in-place activities, and is known for its lower frame rate. The authors also used an ExPose-derived version (HumanAct12-Xpose).
  • UESTC: Features 40 actions, primarily in-place.
  • Human3.6M: A dataset with 32 joints and high intra-category diversity, though it lacks detailed finger joints.

For datasets lacking detailed finger joints (UESTC, Human3.6M), the finger-specific components were omitted. For datasets without locomotion (HumanAct12, UESTC), global body trajectory encoders/decoders were removed.

Key Insights and Ideas

  • Finger Detail Matters: Explicitly modeling fine-grained finger joints is critical for generating realistic human motion, especially for actions involving hand gestures. This is a significant improvement over methods that rely on coarse pose representations.
  • Scalability Through Decoupling: Separating local (joint-level) and global (trajectory) motion, and further decoupling hand and body components, allows the model to scale to diverse datasets and handle complex scenarios like multi-actor interactions.
  • Spatio-Temporal Blocks are Key: The novel ST blocks effectively capture both short-term temporal dynamics and long-term dependencies, leading to more coherent and realistic motion synthesis, particularly for challenging datasets with high variance or low frame rates.
  • Variable Duration is Achievable: Explicitly modeling sequence length as a separate output enables the generation of actions of arbitrary durations, a crucial feature for practical applications.

Key Results

DSAG demonstrates state-of-the-art performance across multiple datasets and evaluation metrics:

  • Superior Quality: DSAG consistently outperforms baseline methods like MUGL, ACTOR, and action2motion in terms of generative quality metrics such as FID, MMD-A, MMD-S, DS, and MS. For instance, on NTU-Xpose-Multi-Person, DSAG achieves significantly lower FID scores compared to MUGL and ACTOR.
  • Improved Finger Realism: The inclusion of dedicated finger joint representations leads to visibly better hand movements in generated sequences, as highlighted in qualitative comparisons.
  • Dataset Generalization: DSAG shows improved generalization capabilities across datasets with varying frame rates, joint counts, and action categories.
  • Handling Diversity: The framework effectively handles datasets with high intra-class diversity (e.g., Human3.6M) and low frame rates (e.g., HumanAct12) due to the specialized ST blocks.

Limitations and Future Work

  • Computational Cost: While DSAG is designed for scalability, training deep generative models on large datasets remains computationally intensive.
  • Dataset Limitations: The quality of generated motion is still dependent on the quality of the input datasets. Datasets with inherent noise or missing information (e.g., Human3.6M's lack of finger data) pose challenges.
  • Future Work: The authors suggest exploring more sophisticated interaction modeling for multi-actor scenarios and investigating the use of these models for tasks like motion editing or retargeting. Further research into more efficient ST block designs and exploring different latent space representations could also be beneficial.

Summary updated

BibTeX
@inproceedings{gupta2023dsag,
  author = {Debtanu Gupta and Shubh Maheshwari and Sai Shashank Kalakonda and Manasvi Vaidyula and Ravi Kiran Sarvadevabhatla},
  title = {{DSAG: A Scalable Deep Framework for Action-Conditioned Multi-Actor Full Body Motion Synthesis}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure