OmniMoGen: Unifying Human Motion Generation via Learning from Interleaved Text-Motion Instructions

Abstract

OmniMoGen is a unified framework that treats human motion generation as a large language model (LLM) task by representing motion as discrete tokens interleaved with text. By training on a massive new dataset of 137K instructions and employing reinforcement learning for physical realism, it achieves state-of-the-art results across text-to-motion, editing, and complex multi-step generation tasks.

Introduction

In natural language processing, models like ChatGPT have unified diverse tasks into a single instruction-following interface. In contrast, human motion generation has remained fragmented, with "siloed" models specialized for text-to-motion, style transfer, or in-betweening. This fragmentation leads to rigidity, where models cannot follow complex, free-form instructions, and architectural bloat, as new capabilities require new modules.

Expand

Source, p. 1

OmniMoGen addresses these hurdles by serving as a "ChatGPT for motion," providing a single architecture that handles any motion task through interleaved text-motion instructions.

Methodology

OmniMoGen treats motion generation as a sequence-to-sequence problem using two core components:

1. Motion Tokenization (RVQ-VAE): A Residual Vector Quantized Variational Autoencoder compresses continuous 3D motion into discrete tokens, creating a "vocabulary" of movement.
2. Autoregressive Transformer (Gemma2-2B): A pre-trained LLM backbone processes interleaved text and motion tokens, predicting the next token in the sequence.

Expand

Source, p. 4

Training Pipeline

The model is trained in two stages. First, Supervised Fine-Tuning (SFT) on the new X2Mo dataset teaches basic instruction following. Second, Reinforcement Learning (GRPO) optimizes for semantic correctness and physical plausibility, specifically penalizing artifacts like foot sliding. Additionally, the OmniMoGen-Think variant enables self-reflection, where the model critiques and refines its own generated motions using a Chain-of-Thought process.

Datasets and Benchmarks

The authors introduced X2Mo, a large-scale dataset of 137K interleaved text-motion instructions covering in-context generation, editing, and reflection. They also developed AnyContext, a benchmark for evaluating interleaved instructions. Evaluations were also conducted on standard benchmarks like HumanML3D and MotionFix.

Key Results and Insights

OmniMoGen achieves state-of-the-art (SOTA) performance:

• Performance: It outperforms previous leaders on HumanML3D and MotionFix, beating the second-best method on MotionFix by 5.3% in retrieval accuracy.
• Interleaved Generation: On the AnyContext benchmark, OmniMoGen achieved an average R@1 of 37.5%, significantly higher than baselines.
• Physical Realism: The model's physical plausibility score (0.97) nearly matches real-world data (0.98).
• Emergent Knowledge: Leveraging its LLM backbone, the model can generate motions based on world knowledge, such as Cristiano Ronaldo’s "SIU" celebration, even without explicit training labels.

Conclusion

OmniMoGen demonstrates that unifying motion generation through interleaved text-motion tokens and LLM-style training is highly effective. By combining large-scale instruction tuning with reinforcement learning for physics, it provides a flexible and realistic foundation for human motion synthesis.

@misc{bu2025omnimogen,
  author = {Wendong Bu and Kaihang Pan and Yuze Lin and Jiacheng Li and Kai Shen and Wenqiao Zhang and Juncheng Li and Jun Xiao and Siliang Tang},
  title = {{OmniMoGen: Unifying Human Motion Generation via Learning from Interleaved Text-Motion Instructions}},
  year = {2025},
  eprint = {2512.19159},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.19159},
}