Abstract
Generating realistic 3D human motion from text requires precise control over style, trajectory, and action. Current models often struggle with multi-layered prompts or unseen attributes, frequently prioritizing one condition over others. We present ACMo, a modular framework that decouples motion attributes and utilizes Large Language Models (LLMs) as reasoning planners. By mapping natural language to a "Local Wordbag" and employing a lightweight Motion Adapter for rapid fine-tuning, ACMo achieves state-of-the-art performance on benchmarks like HumanML3D and effectively handles complex, unseen motion patterns.
Introduction
The field of text-to-motion (T2M) generation has seen significant progress, yet achieving fine-grained, multi-attribute control remains a challenge. Users often require specific movements, such as "walking clockwise in a circle while acting like an elderly person." Existing models face a "precision paradox": they either fail to generalize to unseen descriptions or lose accuracy when multiple constraints are applied.
ACMo addresses these hurdles by decoupling motion attributes—style, trajectory, and action—and using an LLM to bridge the gap between human intent and the model's learned vocabulary.
Methodology
The ACMo framework consists of three primary components designed to provide modular and precise control.
1. Attribute Diffusion Model (ADM)
At its core, ACMo uses a diffusion model that decouples text and motion learning. Unlike object-centric CLIP-based models, ACMo utilizes T5 as a text encoder to better capture action nuances. It also introduces VAE-7, a motion variational autoencoder with a 7-dimensional latent space, preserving higher kinematic detail than previous 4D or 5D versions.
2. LLM Planner
To handle "unseen" attributes (e.g., "walking when his paper is accepted"), an LLM (such as DeepSeek-R1 or GPT-4o) acts as a reasoning planner. It translates complex user prompts into a "Local Wordbag"—a set of 512 high-frequency terms the model understands (e.g., "walking happily, relaxed shoulders").
3. Motion Adapter & Trajectory Control
For rapid adaptation to new styles (e.g., "superman fly"), a lightweight Motion Adapter is used. This allows the system to learn new patterns in approximately 15–20 minutes by fine-tuning only the cross-attention layers. Additionally, a Trajectory ControlNet enables users to specify exact paths for the generated motion.
Datasets and Evaluation
The model was evaluated on several key datasets:
- HumanML3D: The standard benchmark for text-to-motion.
- 100STYLE: Used for evaluating diverse motion style transfer.
- MotionIT: A new dataset introduced by the authors containing 5,000 high-level, goal-driven descriptions to test the LLM Planner's reasoning capabilities.
Results and Discussion
ACMo achieves state-of-the-art results across multiple metrics. On HumanML3D, it reached a Frechet Inception Distance (FID) of 0.102, significantly outperforming the baseline MLD (0.473). The model also showed superior alignment and diversity, with a MultiModality score of 2.614.
The LLM Planner proved highly effective, with DeepSeek-R1 achieving a 75% Rule Consistency Score. This confirms that leveraging world knowledge from LLMs can successfully map abstract human emotions or situations to concrete physical motions.
Conclusion
ACMo provides a robust solution for attribute-controllable motion generation. By decoupling attributes and incorporating an LLM-based planning stage, it overcomes the limitations of data-entrenched models. While challenges remain in long-form narrative generation and occasional "foot skating" during cross-dataset fine-tuning, ACMo represents a significant step toward intuitive and precise 3D human motion synthesis.
Summary updated