UniMo: Unified Motion Generation and Understanding with Chain of Thought

Guocun Wang, Kenkun Liu, Jing Lin, Guorui Song, Jian Li, Xiaoguang Han

TL;DR

UniMo is a unified framework that bridges the gap between 3D human motion generation and text-based motion understanding using Chain-of-Thought (CoT) reasoning. By training an LLM with video-grounded, physically consistent CoT data and optimizing it via Group Relative Policy Optimization (GRPO), UniMo mitigates the cumulative errors typical of standard next-token prediction, achieving state-of-the-art performance in both text-to-motion and motion-to-text tasks.

What Problem Does This Paper Address?

Human motion generation (Text-to-Motion, or T2M) and motion understanding (Motion-to-Text, or M2T) are inherently inverse problems. While they should ideally reinforce each other, previous methods typically treat them as isolated tasks. Recent attempts to unify them using Large Language Models (LLMs) have underperformed due to two primary challenges:

  1. Weak Semantic Mapping: Text descriptions of motion are often coarse and simple, meaning a single sentence can correspond to many different physical movements.
  2. Causality Mismatch: Unlike natural language, human motion sequences possess weaker sequential causality. Applying the standard autoregressive "next-token prediction" paradigm to motion tokens leads to rapid accumulation of prediction errors over time.

Additionally, prior attempts to introduce CoT reasoning to motion models generated step-by-step text descriptions purely from existing text captions. This text-only approach often resulted in "hallucinated" reasoning steps that did not actually match the physical realities of the underlying 3D motion.

Methods

UniMo addresses these challenges through a two-stage training pipeline (Supervised Fine-Tuning followed by Reinforcement Learning) built on top of the open-source Qwen2.5-3B-Instruct model.

Figure 1: The UniMo is trained in two stages: the SFT stage and the reinforcement learning stage with GRPO. In the SFT stage, the model is teached to perform both T2M and M2T tasks with structured reasoning, i.e. CoT. In the RL stage, the model is further optimized with task-specific rewards, enabling unified and interpretable motion generation and understanding.

1. Motion Tokenization

To allow the LLM to process 3D human motion, continuous motion sequences (represented as 3D joint positions over time) are discretized into discrete motion tokens using a pretrained Vector Quantized Variational Autoencoder (VQ-VAE).

2. Video-Grounded CoT Data Curation

To create accurate, physically consistent reasoning traces, the authors developed a multimodal data curation pipeline. They rendered 3D motion sequences into 2D videos using the Blender 3D engine. They then prompted a powerful Vision-Language Model (Qwen2.5-VL-72B) with both the video and the original text caption to generate highly accurate, chronologically ordered, step-by-step reasoning traces enclosed in <think> tags.

Figure 2: Illustration of the CoT annotation process. Human joint sequences are rendered in the Blender and paired with captions, which are further processed by the Qwen2.5-VL- 72B to generate reasoning traces.

3. CoT-Guided Unified Modeling (SFT Stage)

During Supervised Fine-Tuning, the model learns to perform both tasks sequentially. For T2M, given a text prompt, the model first outputs a step-by-step motion plan (CoT) before generating the motion tokens. For M2T, the model ingests motion tokens, generates a CoT breakdown of the action, and then outputs the final caption.

4. Post-Training via GRPO (RL Stage)

To overcome the limitations of next-token prediction, the authors employ Group Relative Policy Optimization (GRPO). Instead of evaluating token-by-token, GRPO samples a group of completions for a single prompt and optimizes the policy based on relative rewards across the group. The model is optimized using four task-specific rewards:

  • Format Reward: Enforces strict adherence to the <think> and <Motion>/<Answer> structure.
  • Motion Similarity Reward: Computes the cosine similarity between the generated motion and ground truth using a pretrained motion encoder.
  • Semantic Similarity Reward: Measures cross-modal alignment between generated motion tokens and the input text prompt.
  • Caption Similarity Reward: Uses a CLIP text encoder to evaluate the semantic alignment of generated captions against reference captions.

Datasets

The framework is trained and evaluated on HumanML3D, a large-scale 3D human motion-language benchmark containing 14,616 motion clips and 44,970 textual descriptions. The authors augmented this dataset by generating corresponding video-grounded CoT annotations for every motion-caption pair.

Key Insights and Ideas

  • Bidirectional Synergy: Jointly training T2M and M2T tasks creates a powerful cross-modal loop. The ablation studies show that unified modeling outperforms single-task training across all metrics, proving that learning to describe a motion helps the model generate it better, and vice versa.
  • Grounding CoT in Vision: Generating CoT data from rendered videos rather than text rewrites ensures that the language model's internal reasoning aligns perfectly with physical constraints and spatial dynamics, drastically reducing hallucinations.
  • Group Optimization for Sequences: Using GRPO to optimize entire sequences based on global similarity rewards effectively bypasses the compounding error problem inherent to autoregressive next-token prediction in non-linguistic domains.

Key Results

UniMo achieves state-of-the-art performance on both tasks simultaneously, outperforming specialized single-task models.

Text-to-Motion (T2M)

On the HumanML3D evaluation, UniMo achieves a Top-1 R-Precision of 0.539 and a remarkably low Fréchet Inception Distance (FID) of 0.177, outperforming prominent diffusion-based models (like MLD and MotionDiffuse) and autoregressive models (like Motion-R1).

Motion-to-Text (M2T)

For motion captioning, UniMo sets a new benchmark, substantially outperforming previous unified models like MotionLLM:

  • BLEU@1: 63.10 (vs. MotionLLM's 54.53)
  • CIDEr: 46.69 (vs. MotionLLM's 33.74)
  • BertScore: 54.26 (vs. MotionLLM's 42.63)

Qualitative results highlight that UniMo generates highly coherent, sequential actions that strictly follow complex user instructions.

Qualitative text-to-motion generation comparisons.

Limitations and Future Work

The authors do not explicitly include a dedicated limitations section in the paper. However, inherent limitations of this architecture include:

  • Computational Overhead: The requirement to render 3D joint sequences into videos via a 3D engine (Blender) to curate the CoT dataset introduces a heavy preprocessing pipeline that may be difficult to scale to massive, unlabelled motion datasets.
  • Tokenizer Dependency: The performance remains bounded by the discrete codebook capacity of the frozen VQ-VAE motion tokenizer.

Future work could explore end-to-end tokenization strategies or extending this unified CoT framework to multi-person interactions and longer, compositional motion sequences.

Summary updated

Abstract

Existing 3D human motion generation and understanding methods often exhibit limited interpretability, restricting effective mutual enhancement between these inherently related tasks. While current unified frameworks based on large language models (LLMs) leverage linguistic priors, they frequently encounter challenges in semantic alignment and task coherence. Moreover, the next-token prediction paradigm in LLMs is ill-suited for motion sequences, causing cumulative prediction errors. To address these limitations, we propose UniMo, a novel framework that integrates motion-language information and interpretable chain of thought (CoT) reasoning into the LLM via supervised fine-tuning (SFT). We further introduce reinforcement learning with Group Relative Policy Optimization (GRPO) as a post-training strategy that optimizes over groups of tokens to enforce structural correctness and semantic alignment, mitigating cumulative errors in motion token prediction. Extensive experiments demonstrate that UniMo significantly outperforms existing unified and task-specific models, achieving state-of-the-art performance in both motion generation and understanding.
BibTeX
@inproceedings{wang2026unimo,
  author = {Guocun Wang and Kenkun Liu and Jing Lin and Guorui Song and Jian Li and Xiaoguang Han},
  title = {{UniMo: Unified Motion Generation and Understanding with Chain of Thought}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2026},
  doi = {10.1609/aaai.v40i12.37936},
}
Figure