AMD: Autoregressive Motion Diffusion

Bo Han, Hao Peng, Minjing Dong, Chang Xu, Yi Ren, Yixuan Shen, Yuheng LiarXiv 2023 · DOI · arXiv

Abstract

Generating long, coherent 3D human motions from complex textual descriptions remains a significant challenge in computer animation. Existing text-to-motion models often struggle with temporal consistency and physical realism when faced with multi-stage prompts. We present Autoregressive Motion Diffusion (AMD), a framework that combines the high-fidelity generation of diffusion models with the global structural benefits of autoregressive modeling. To facilitate research in long-form motion synthesis, we introduce HumanLong3D, the first large-scale dataset featuring extensive textual descriptions paired with complex 3D motions. Our results demonstrate that AMD achieves state-of-the-art performance in generating diverse and physically plausible long-duration sequences.

Introduction

Recent advancements in Text-to-Motion (T2M) synthesis have enabled the generation of realistic human movements from simple prompts. However, models like MDM or MLD often fail when tasked with "compound prompts"—sequences of actions such as "a person walking, then sitting down, then jumping up." These failures typically manifest as "stiff" limbs, loss of temporal coherence, or premature termination of the action sequence.

The authors identify two primary bottlenecks in the current landscape:

  1. Data Scarcity: Existing datasets like HumanML3D primarily contain short, isolated actions, lacking the complexity required for long-form synthesis.
  2. Mapping Complexity: The many-to-many mapping between nuanced, long-form sentences and high-dimensional temporal motion sequences is mathematically difficult for standard diffusion models to maintain over extended durations.

The AMD Framework

The core innovation of AMD is its hybrid architecture. Rather than generating a long motion sequence in a single pass, AMD adopts an iterative approach that balances local quality with global structure.

The AMD framework utilizes a context encoder to maintain temporal consistency across generated motion segments.

Autoregressive Iteration and Diffusion

AMD generates motion in discrete segments. For each new segment, the model conditions its generation on the previous motion clip and the preceding text prompt. This autoregressive loop ensures seamless transitions between distinct actions. Within each segment, a Diffusion Module denoises a random signal into a high-fidelity motion clip, leveraging the generative power of diffusion processes to create realistic movement.

Duration Prediction and Geometric Constraints

To handle variable action lengths, AMD includes a dedicated network that treats motion duration as a probability density estimation problem. This allows the model to adapt to prompts requiring different timeframes (e.g., a "quick jump" vs. a "slow stroll"). Additionally, explicit loss functions for joint position, velocity, rotation, and foot contact are employed to prevent common artifacts like foot sliding or disjointed skeletal structures.

Datasets: HumanLong3D

A major contribution of this work is the release of HumanLong3D. This dataset contains 158,179 textual descriptions paired with 43,696 complex 3D motions, providing the necessary scale and temporal labels for training long-form models. The authors also introduce HumanMusic, a dataset for Audio-to-Motion synthesis featuring over 137,000 paired dance and music samples.

Key Results

AMD sets a new state-of-the-art (SOTA) on the HumanLong3D benchmark. It achieved a Fréchet Inception Distance (FID) of 0.225, significantly outperforming previous models like T2M-GPT (1.249) and MLD (3.843). A lower FID indicates that the generated motions are statistically much closer to real human data.

Qualitative comparison showing AMD's superior temporal coherence and realism compared to baseline models.

Qualitatively, AMD avoids the "limb stiffness" and "premature termination" issues prevalent in earlier models. It maintains high diversity (4.515), demonstrating its ability to generate multiple valid interpretations of a single complex prompt without "averaging out" the motion.

Conclusion and Future Work

AMD demonstrates that combining autoregressive structures with diffusion models effectively addresses the challenges of long-form motion synthesis. While the model excels in text-to-motion, future work will focus on improving beat alignment in music-to-dance tasks—where it currently ranks second to reinforcement learning-based models—and expanding to additional input modalities like 2D video guidance or specific skeletal constraints.

Summary updated

BibTeX
@misc{han2023amd,
  author = {Bo Han and Hao Peng and Minjing Dong and Chang Xu and Yi Ren and Yixuan Shen and Yuheng Li},
  title = {{AMD: Autoregressive Motion Diffusion}},
  year = {2023},
  eprint = {2305.09381},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.09381},
}
Figure