Abstract
Conditional human motion synthesis (HMS) aims to generate human motion sequences that conform to specific conditions. Text and audio represent the two predominant modalities employed as HMS control conditions. While existing research has primarily focused on single conditions, the multi-condition human motion synthesis remains underexplored. In this study, we propose a multi-condition HMS framework, termed MCM, based on a dual-branch structure composed of a main branch and a control branch. This framework effectively extends the applicability of the diffusion model, which is initially predicated solely on textual conditions, to auditory conditions. This extension encompasses both music-to-dance and co-speech HMS while preserving the intrinsic quality of motion and the capabilities for semantic association inherent in the original model. Furthermore, we propose the implementation of a Transformer-based diffusion model, designated as MWNet, as the main branch. This model adeptly apprehends the spatial intricacies and inter-joint correlations inherent in motion sequences, facilitated by the integration of multi-wise self-attention modules. Extensive experiments show that our method achieves competitive results in single-condition and multi-condition HMS tasks.
Introduction
Generating realistic 3D human motion is a cornerstone of modern animation and game development. However, current AI models often suffer from "one-track minds," typically specializing in either text-to-motion (e.g., "a person doing a somersault") or audio-to-motion (e.g., dancing to a specific beat), but rarely both simultaneously.
The MCM framework addresses three primary challenges in multi-condition synthesis:
- Data Scarcity: High-quality datasets containing aligned text, audio, and 3D motion are extremely rare.
- Alignment Issues: Text and audio operate on different temporal scales, making it difficult to synchronize semantic meaning with rhythmic beats.
- Catastrophic Forgetting: Fine-tuning a text-based model on audio data often causes it to lose its original semantic understanding.
The MCM Framework
To overcome these hurdles, the researchers developed a dual-branch architecture inspired by ControlNet. This structure allows the model to learn new modalities without sacrificing its existing knowledge.
Dual-Branch Architecture
- Main Branch: This branch consists of a pre-trained text-to-motion model. Crucially, its weights are frozen during the training of new modalities. This ensures the model retains its "semantic common sense" and high-quality motion generation capabilities.
- Control Branch: This is a trainable copy of the main branch designed to process the secondary modality, such as audio features extracted via Jukebox.
- Bridge Modules: These modules connect the two branches using "zero-initialization." Initially, they contribute nothing to the output, but as training progresses, they learn to "nudge" the main branch's motion to align with the audio rhythm.
MWNet: Spatial-Aware Motion Encoding
A key innovation within the main branch is MWNet (Multi-Wise Network). The authors argue that standard models focus too heavily on temporal aspects while neglecting spatial relationships between joints. MWNet incorporates Multi-Wise Attention, which combines standard Time-wise self-attention with Channel-wise self-attention. By treating the channel dimension as spatial information, the model better captures complex inter-joint correlations, leading to more natural and physically plausible movements.
Evaluation and Results
The model was evaluated on major benchmarks including HumanML3D (text), AIST++ (dance), and BEAT (speech). To handle the lack of multi-modal data in AIST++ and BEAT, the authors generated "pseudo-text" labels to create the necessary training pairs.
Performance Metrics
- Motion Quality: On HumanML3D, MWNet achieved a Frechet Inception Distance (FID) of 0.053, significantly outperforming previous state-of-the-art models like MDM (0.544).
- Rhythmic Alignment: In music-to-dance tasks, MCM achieved a Beat Alignment Score of 0.275, proving its superior ability to follow musical rhythms compared to models designed exclusively for dance.
- User Preference: In qualitative testing with unseen music, users preferred MCM’s results over 60% of the time compared to existing fusion methods, noting better synchronization between gestures and audio.
Conclusion
MCM represents a significant step forward in multi-modal human motion synthesis. By leveraging a dual-branch structure and spatial-aware attention, it produces high-quality motion that is both semantically accurate and rhythmically aligned. While the current framework shows moderate diversity in its outputs, future work aims to expand the "creative range" of the model and incorporate additional conditions like scene geometry.
Summary updated