HumanDreamer: Generating Controllable Human-Motion Videos via Decoupled Generation

Boyuan Wang, Xiaofeng Wang, Chaojun Ni, Guosheng Zhao, Zhiqin Yang, Zheng Zhu, Muyang Zhang, Yukun Zhou, Xinze Chen, Guan Huang, Lihong Liu, Xingang Wang

Abstract

Human-motion video generation has been a challenging task, primarily due to the difficulty inherent in learning human body movements. While some approaches have attempted to drive human-centric video generation explicitly through pose control, these methods typically rely on poses derived from existing videos, thereby lacking flexibility. To address this, we propose HumanDreamer, a decoupled human video generation framework that first generates diverse poses from text prompts and then leverages these poses to generate human-motion videos. Specifically, we propose MotionVid, the largest dataset for human-motion pose generation. Based on the dataset, we present MotionDiT, which is trained to generate structured human-motion poses from text prompts. Besides, a novel LAMA loss is introduced, which together contribute to a significant improvement in FID by 62.4%, along with respective enhancements in R-precision for top1, top2, and top3 by 41.8%, 26.3%, and 18.3%, thereby advancing both the Text-to-Pose control accuracy and FID metrics. Our experiments across various Pose-to-Video baselines demonstrate that the poses generated by our method can produce diverse and high-quality human-motion videos. Furthermore, our model can facilitate other downstream tasks, such as pose sequence prediction and 2D-3D motion lifting.

Introduction

Generating realistic human motion from text is notoriously difficult. Direct text-to-video models often struggle with "hallucinated" limbs, distorted bodies, or movements that do not accurately match the prompt. While some researchers use "pose-guided" generation (where a skeleton drives the video), they usually extract those skeletons from existing videos. This creates a "chicken and egg" problem: one cannot generate a novel movement without an existing video of it. HumanDreamer bridges this gap by creating a reliable way to generate entirely new, physically plausible pose sequences from scratch using only a text description.

The HumanDreamer Framework

The core innovation of HumanDreamer is its decoupled approach. By splitting the generation process into two distinct stages, the model avoids the complexity of learning geometry and appearance simultaneously.

The HumanDreamer framework decouples video generation into text-to-pose and pose-to-video stages.

1. Text-to-Pose (MotionDiT)

Instead of pixels, the model first generates a sequence of 2D keypoints. This is handled by MotionDiT, a Diffusion Transformer architecture tailored for structured human motion.

The MotionDiT architecture utilizes Local Feature Aggregation and Global Attention blocks to generate structured poses.

MotionDiT incorporates several key components:

  • Local Feature Aggregation: Ensures adjacent joints (like an elbow and a wrist) move in a correlated, physically sensible way.
  • Global Attention: Maintains consistency across the entire duration of the motion sequence.
  • LAMA Loss (Latent Semantic Alignment): A novel loss function that aligns the model's internal latent features with a pre-trained text-motion encoder (CLoP), ensuring the generated action strictly follows the text prompt.

2. Pose-to-Video

Once the skeleton sequence is generated, a second model (based on Stable Video Diffusion) renders the final video. The skeleton acts as a rigid guide, ensuring character proportions and movements remain stable throughout the clip.

Data Curation: MotionVid

The backbone of this research is MotionVid, a massive new dataset curated to support high-quality motion learning. It contains 1.2 million text-pose-video pairs, significantly larger than previous benchmarks like HumanML3D. The data was sourced from 13 public datasets and 3.4 million internet videos, subjected to a rigorous cleaning pipeline that filtered for aesthetic quality, face visibility, and significant movement.

Key Results

HumanDreamer demonstrates substantial quantitative and qualitative improvements:

  • Text-to-Pose Accuracy: Improved FID (realism and diversity) by 62.4% over previous state-of-the-art models.
  • Prompt Adherence: R-precision (text-motion matching) increased by 41.8% for top-1 results.
  • Video Quality: Compared to general video models like Mochi1 and CogVideoX, HumanDreamer produces videos with fewer body distortions and smoother transitions.
  • Versatility: The generated 2D poses are high-quality enough to be "lifted" into 3D or used for downstream tasks like motion prediction.

Conclusion and Future Work

By decoupling geometry from appearance, HumanDreamer achieves unprecedented control and realism in human-centric video generation. While currently focused on single-person scenes, future work will explore complex multi-person interactions and bridging the 2D-to-3D gap for even more complex maneuvers.

Summary updated

Abstract

Human-motion video generation has been a challenging task, primarily due to the difficulty inherent in learning human body movements. While some approaches have attempted to drive human-centric video generation explicitly through pose control, these methods typically rely on poses derived from existing videos, thereby lacking flexibility. To address this, we propose HumanDreamer, a decoupled human video generation framework that first generates diverse poses from text prompts and then leverages these poses to generate human-motion videos. Specifically, we propose MotionVid, the largest dataset for human-motion pose generation. Based on the dataset, we present MotionDiT, which is trained to generate structured human-motion poses from text prompts. Besides, a novel LAMA loss is introduced, which together contribute to a significant improvement in FID by 62.4%, along with respective enhancements in R-precision for top1, top2, and top3 by 41.8%, 26.3%, and 18.3%, thereby advancing both the Text-to-Pose control accuracy and FID metrics. Our experiments across various Pose-to-Video baselines demonstrate that the poses generated by our method can produce diverse and high-quality human-motion videos. Furthermore, our model can facilitate other downstream tasks, such as pose sequence prediction and 2D-3D motion lifting.
BibTeX
@inproceedings{wang2025humandreamer,
  author = {Boyuan Wang and Xiaofeng Wang and Chaojun Ni and Guosheng Zhao and Zhiqin Yang and Zheng Zhu and Muyang Zhang and Yukun Zhou and Xinze Chen and Guan Huang and Lihong Liu and Xingang Wang},
  title = {{HumanDreamer: Generating Controllable Human-Motion Videos via Decoupled Generation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=aGXVPQKLct},
}
Figure