MCM: Multi-condition Motion Synthesis Framework

Zeyu Ling, Bo Han, Yongkang Wong, Han Lin

Abstract

Generating high-fidelity 3D human motion from multiple simultaneous conditions, such as text descriptions and audio (music or speech), is a complex task due to data scarcity and modality misalignment. We introduce MCM (Multi-Condition Motion), a novel framework presented at IJCAI-24. MCM utilizes a dual-branch diffusion architecture and a "Multi-Wise" attention mechanism to integrate non-aligned modalities effectively. By freezing a pre-trained text-to-motion branch and employing a trainable control branch for audio, the model avoids catastrophic forgetting while achieving state-of-the-art performance in both text-to-motion and music-to-dance tasks.

Introduction

The synthesis of human motion from AI typically relies on a single input modality, such as a text prompt ("a person dancing") or an audio file. However, real-world applications often require multi-modal control—for instance, generating a dance that follows a specific stylistic description while remaining synchronized with a musical beat. Integrating these disparate signals is difficult because text and audio are temporally non-aligned and high-quality multi-modal datasets are scarce.

Visual examples of motion generated by MCM from diverse text and audio prompts, demonstrating semantic and rhythmic alignment.

Methodology

MCM addresses these challenges through a dual-branch diffusion structure designed to preserve semantic intelligence while incorporating new rhythmic constraints.

Dual-Branch Architecture

The framework consists of two primary components:

  1. Main Branch (MWNet): A Transformer-based Denoising Diffusion Probabilistic Model (DDPM) pre-trained on text-to-motion tasks. To maintain its semantic understanding, this branch is "frozen" during multi-modal training.
  2. Control Branch: Inspired by ControlNet, this branch mirrors the main branch but is trained to process audio features (e.g., via a Jukebox encoder). It "nudges" the generation process to align with the audio rhythm.
  3. Bridge Modules: These modules connect the two branches, allowing the control branch to influence the main branch's output without overwriting its learned text-motion associations.
The MCM dual-branch architecture, illustrating the flow from text and audio inputs through the Main and Control branches.

Multi-Wise Attention

A key innovation within MWNet is the Multi-Wise attention mechanism, which processes motion data in two distinct ways:

  • Time-wise Attention: Captures the temporal sequence and flow of movement.
  • Channel-wise Attention: Analyzes the spatial correlations between different body joints. This focus on spatial relationships significantly improves the physical realism of the generated motion.

Experiments and Results

The researchers evaluated MCM on standard benchmarks including HumanML3D (text-to-motion), AIST++ (music-to-dance), and BEAT (co-speech motion). To overcome the lack of multi-modal data, they employed pseudo-labeling to create text-audio pairs for training.

Key Performance Metrics

  • Text-to-Motion: On HumanML3D, MCM achieved a Frechet Inception Distance (FID) of 0.053, outperforming previous models like MDM (0.544) and MotionDiffuse (0.630).
  • Music-to-Dance: MCM reached a top $FID_k$ score of 15.57 on AIST++, surpassing specialized models like EDGE.
  • User Preference: In comparative studies, users preferred MCM-generated motions over 90% of the time regarding beat alignment and text matching.

Conclusion

MCM demonstrates that a "freeze and control" strategy, combined with spatial-aware attention mechanisms, can successfully bridge the gap between text and audio for motion synthesis. While the inclusion of channel-wise attention slightly impacts motion diversity, the gains in physical realism and semantic accuracy mark a significant step forward for multi-modal generative AI.

Summary updated

BibTeX
@inproceedings{ling2024mcm,
  author = {Zeyu Ling and Bo Han and Yongkang Wong and Han Lin},
  title = {{MCM: Multi-condition Motion Synthesis Framework}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=WmGRi0QajS},
}
Figure