ACMo: Attribute Controllable Motion Generation

Mingjie Wei, Xuemei Xie, Guangming ShiarXiv 2025 · DOI · arXiv

Abstract

Generating realistic 3D human motion from text requires precise control over style, trajectory, and action. Current models often struggle with multi-layered prompts or unseen attributes, frequently prioritizing one condition over others. We present ACMo, a modular framework that decouples motion attributes and utilizes Large Language Models (LLMs) as reasoning planners. By mapping natural language to a "Local Wordbag" and employing a lightweight Motion Adapter for rapid fine-tuning, ACMo achieves state-of-the-art performance on benchmarks like HumanML3D and effectively handles complex, unseen motion patterns.

Introduction

The field of text-to-motion (T2M) generation has seen significant progress, yet achieving fine-grained, multi-attribute control remains a challenge. Users often require specific movements, such as "walking clockwise in a circle while acting like an elderly person." Existing models face a "precision paradox": they either fail to generalize to unseen descriptions or lose accuracy when multiple constraints are applied.

Example of attribute control where a character follows a circular trajectory while maintaining a specific style.

ACMo addresses these hurdles by decoupling motion attributes—style, trajectory, and action—and using an LLM to bridge the gap between human intent and the model's learned vocabulary.

Methodology

The ACMo framework consists of three primary components designed to provide modular and precise control.

The ACMo framework overview, illustrating the integration of Motion Prompts, Text Prompts via an LLM Planner, and Trajectory ControlNet.

1. Attribute Diffusion Model (ADM)

At its core, ACMo uses a diffusion model that decouples text and motion learning. Unlike object-centric CLIP-based models, ACMo utilizes T5 as a text encoder to better capture action nuances. It also introduces VAE-7, a motion variational autoencoder with a 7-dimensional latent space, preserving higher kinematic detail than previous 4D or 5D versions.

2. LLM Planner

To handle "unseen" attributes (e.g., "walking when his paper is accepted"), an LLM (such as DeepSeek-R1 or GPT-4o) acts as a reasoning planner. It translates complex user prompts into a "Local Wordbag"—a set of 512 high-frequency terms the model understands (e.g., "walking happily, relaxed shoulders").

3. Motion Adapter & Trajectory Control

For rapid adaptation to new styles (e.g., "superman fly"), a lightweight Motion Adapter is used. This allows the system to learn new patterns in approximately 15–20 minutes by fine-tuning only the cross-attention layers. Additionally, a Trajectory ControlNet enables users to specify exact paths for the generated motion.

Datasets and Evaluation

The model was evaluated on several key datasets:

  • HumanML3D: The standard benchmark for text-to-motion.
  • 100STYLE: Used for evaluating diverse motion style transfer.
  • MotionIT: A new dataset introduced by the authors containing 5,000 high-level, goal-driven descriptions to test the LLM Planner's reasoning capabilities.

Results and Discussion

ACMo achieves state-of-the-art results across multiple metrics. On HumanML3D, it reached a Frechet Inception Distance (FID) of 0.102, significantly outperforming the baseline MLD (0.473). The model also showed superior alignment and diversity, with a MultiModality score of 2.614.

The LLM Planner proved highly effective, with DeepSeek-R1 achieving a 75% Rule Consistency Score. This confirms that leveraging world knowledge from LLMs can successfully map abstract human emotions or situations to concrete physical motions.

Conclusion

ACMo provides a robust solution for attribute-controllable motion generation. By decoupling attributes and incorporating an LLM-based planning stage, it overcomes the limitations of data-entrenched models. While challenges remain in long-form narrative generation and occasional "foot skating" during cross-dataset fine-tuning, ACMo represents a significant step toward intuitive and precise 3D human motion synthesis.

Summary updated

Abstract

Attributes such as style, fine-grained text, and trajectory are specific conditions for describing motion. However, existing methods often lack precise user control over motion attributes and suffer from limited generalizability to unseen motions. This work introduces an Attribute Controllable Motion generation architecture, to address these challenges via decouple any conditions and control them separately. Firstly, we explored the Attribute Diffusion Model to imporve text-to-motion performance via decouple text and motion learning, as the controllable model relies heavily on the pre-trained model. Then, we introduce Motion Adpater to quickly finetune previously unseen motion patterns. Its motion prompts inputs achieve multimodal text-to-motion generation that captures user-specified styles. Finally, we propose a LLM Planner to bridge the gap between unseen attributes and dataset-specific texts via local knowledage for user-friendly interaction. Our approach introduces the capability for motion prompts for stylize generation, enabling fine-grained and user-friendly attribute control while providing performance comparable to state-of-the-art methods. Project page: https://mjwei3d.github.io/ACMo/
BibTeX
@misc{wei2025acmo,
  author = {Mingjie Wei and Xuemei Xie and Guangming Shi},
  title = {{ACMo: Attribute Controllable Motion Generation}},
  year = {2025},
  eprint = {2503.11038},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.11038},
}
Figure