HumanDreamer: Generating Controllable Human-Motion Videos via Decoupled Generation

Boyuan Wang, Xiaofeng Wang, Chaojun Ni, Guosheng Zhao, Zhiqin Yang, Zheng Zhu, Muyang Zhang, Yukun Zhou, Xinze Chen, Guan Huang, Lihong Liu, Xingang Wang

Abstract

Human-motion video generation has been a challenging task, primarily due to the difficulty inherent in learning human body movements. While some approaches have attempted to drive human-centric video generation explicitly through pose control, these methods typically rely on poses derived from existing videos, thereby lacking flexibility. To address this, we propose HumanDreamer, a decoupled human video generation framework that first generates diverse poses from text prompts and then leverages these poses to generate human-motion videos. Specifically, we propose MotionVid, the largest dataset for human-motion pose generation. Based on the dataset, we present MotionDiT, which is trained to generate structured human-motion poses from text prompts. Besides, a novel LAMA loss is introduced, which together contribute to a significant improvement in FID by 62.4%, along with respective enhancements in R-precision for top1, top2, and top3 by 41.8%, 26.3%, and 18.3%. Our experiments demonstrate that the poses generated by our method can produce diverse and high-quality human-motion videos, facilitating downstream tasks such as pose sequence prediction and 2D-3D motion lifting.

The Challenge of Human Motion

Generating realistic videos of humans performing specific actions is notoriously difficult. Direct text-to-pixel models often struggle with "hallucinations," resulting in distorted limbs, fragmented movements, or a lack of physical consistency. While existing methods use pose guidance to mitigate these issues, they typically require a pre-existing video to extract those poses, which limits creativity—you can only generate what you already have a reference for. The authors identify a fundamental challenge: directly mapping abstract text prompts to complex, high-dimensional video pixels is too large a leap. HumanDreamer bridges this gap by first translating text into a structured skeletal pose sequence, then "skinning" that sequence into a full video.

Methods: Decoupled Generation

The core of HumanDreamer is its decoupled pipeline, which breaks the generation process into two manageable steps: Text-to-Pose and Pose-to-Video.

Overview of the HumanDreamer decoupled generation pipeline, showing Text-to-Pose followed by Pose-to-Video.

1. Text-to-Pose (MotionDiT)

The authors propose a Motion Diffusion Transformer (MotionDiT). Unlike standard transformers, it includes a Global Attention Block to capture long-range temporal dependencies (the "big picture" of the movement) and a Local Feature Aggregation module to ensure that adjacent joints move realistically in relation to one another.

2. LAMA Loss

To ensure the generated poses actually match the text, they introduced the LAtent seMantic Alignment (LAMA) loss. This uses a pre-trained contrastive model (CLoP) to align the features of the generated motion with the text embedding in a shared latent space.

Detailed architecture of MotionDiT, highlighting the Global Attention Block, Local Feature Aggregation, and the LAMA loss alignment.

3. Pose-to-Video

Once the skeletal sequence is generated, a video diffusion backbone (based on CogVideoX) is used to render the final human appearance, conditioned on the generated poses and an initial reference image.

The MotionVid Dataset

The researchers found that existing datasets were either too small or lacked detail (e.g., missing hands or faces). To solve this, they created MotionVid, the largest dataset of its kind with 1.2 million text-pose pairs. Curated from public datasets and 3.4 million internet videos, the dataset underwent a rigorous cleaning pipeline filtering for movement intensity, human coverage, face visibility, and caption accuracy.

Key Results and Insights

HumanDreamer significantly outperformed existing state-of-the-art (SOTA) motion models like MLD and T2M-GPT:

  • FID (Fréchet Inception Distance): Achieved a 62.4% improvement, indicating much higher distributional similarity to real human motion.
  • R-Precision: Top-1 accuracy in matching text to motion improved by 41.8%.
  • Qualitative Superiority: Compared to direct video generators like Mochi1, HumanDreamer produced videos with smoother transitions, better facial expressions, and fewer body distortions.
  • Versatility: The model proved capable of downstream tasks like pose sequence prediction (filling in missing frames) and 2D-to-3D motion lifting.

The authors argue that the "search space" for Text-to-Pose is much smaller and more structured than Text-to-Pixel. By solving the motion problem in the skeletal domain first, the subsequent video generation task becomes a much easier "rendering" problem.

Limitations and Future Work

While the paper demonstrates impressive results, generating complex interactions between multiple people remains a challenge, as the current Human Quality Filter prioritizes single-person scenes. Additionally, the two-step diffusion process is computationally intensive, though it allows for more efficient testing of the motion component independently of the rendering stage.

Summary updated

Abstract

Human-motion video generation has been a challenging task, primarily due to the difficulty inherent in learning human body movements. While some approaches have attempted to drive human-centric video generation explicitly through pose control, these methods typically rely on poses derived from existing videos, thereby lacking flexibility. To address this, we propose HumanDreamer, a decoupled human video generation framework that first generates diverse poses from text prompts and then leverages these poses to generate human-motion videos. Specifically, we propose MotionVid, the largest dataset for human-motion pose generation. Based on the dataset, we present MotionDiT, which is trained to generate structured human-motion poses from text prompts. Besides, a novel LAMA loss is introduced, which together contribute to a significant improvement in FID by 62.4%, along with respective enhancements in R-precision for top1, top2, and top3 by 41.8%, 26.3%, and 18.3%, thereby advancing both the Text-to-Pose control accuracy and FID metrics. Our experiments across various Pose-to-Video baselines demonstrate that the poses generated by our method can produce diverse and high-quality human-motion videos. Furthermore, our model can facilitate other downstream tasks, such as pose sequence prediction and 2D-3D motion lifting.
BibTeX
@inproceedings{wang2025humandreamer,
  author = {Boyuan Wang and Xiaofeng Wang and Chaojun Ni and Guosheng Zhao and Zhiqin Yang and Zheng Zhu and Muyang Zhang and Yukun Zhou and Xinze Chen and Guan Huang and Lihong Liu and Xingang Wang},
  title = {{HumanDreamer: Generating Controllable Human-Motion Videos via Decoupled Generation}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  url = {https://openreview.net/forum?id=aGXVPQKLct},
}
Figure