Human Motion Aware Text-to-Video Generation with Explicit Camera Control

Taehoon Kim, Chanhee Kang, JaeHyuk Park, Daun Jeong, ChangHee Yang, Suk-Ju Kang, Kyeongbo KongWACV 2024 · OpenReview

TL;DR

Researchers have developed a two-stage framework that generates high-quality videos of human movement from simple text prompts. By first generating a 3D human motion sequence and then projecting it into 2D skeletons with explicit camera controls (like "top view" or "zoom in"), the system provides a structural "blueprint" that prevents common AI video pitfalls like disappearing limbs, incorrect scaling, and jittery motion.

Addressing the Challenges of Human Motion in T2V

Current text-to-video (T2V) models, while impressive, often struggle when asked to depict complex human actions. They typically suffer from three fundamental issues:

  1. Motion Ambiguity: The AI might understand a verb like "punching" but generate a static or anatomically impossible pose.
  2. Scale Problems: Lacking a sense of 3D space, models often produce humans that are disproportionately large or small for their environment.
  3. Temporal Inconsistency: A subject might look correct in the first frame but morph into a different shape or disappear entirely as the video progresses.
Common issues in T2V generation: (a) Motion Ambiguity, (b) Scale Problem, and (c) Temporal Consistency Problem.

While "skeleton-guided" models exist to mitigate these issues, they usually require users to provide a pre-existing skeleton video. This research introduces a method to generate both the motion and the video from scratch using only text, while granting explicit control over the "virtual camera."

The Two-Stage Framework

The proposed pipeline decouples the structural physics of motion from the stylistic rendering of the video.

Stage 1: Text-to-Skeleton with Camera Control

  • Motion Generation: The system processes a text prompt (e.g., "A person is running") using a pre-trained model like T2M-GPT or MDM to generate a sequence of 3D human mesh vertices.
  • Explicit Camera Control: This is the core innovation. Users can provide a secondary camera prompt (e.g., "Tilt Down"). The Motion-to-Skeleton Projection Module applies a mathematical projection matrix to the 3D joints. By adjusting rotation and translation parameters based on the camera prompt, it "films" the 3D motion from the desired angle, outputting a 2D skeleton sequence.

Stage 2: Skeleton-guided Video Generation

The 2D skeletons from Stage 1 act as a frame-by-frame structural guide. This sequence, along with the original text prompt, is fed into a diffusion-based T2V model (such as Follow Your Pose). Because the model has a precise map of where the human should be, it can focus its computational resources on rendering textures and backgrounds rather than attempting to learn human anatomy on the fly.

The proposed two-stage framework featuring Text-to-Skeleton and Pose-Guided Text-to-Video modules.

Key Results and Performance

The framework leverages large-scale human motion datasets like HumanML3D and KIT Motion-Language for training. In comparative evaluations, the proposed method significantly outperformed existing T2V models:

  • Action Classification (AC): The best configuration (Follow Your Pose + the proposed framework) achieved 48.9% accuracy, compared to 32.5% for ModelScope and 34.6% for Zeroscope.
  • Frame Consistency (FC): The model maintained higher visual stability across frames, scoring 92.2% in consistency tests.
  • User Preference: In a study with 100 participants, 56.8% of evaluators preferred this framework over competitors like Runway AI or Zeroscope, citing superior semantic relevance and motion realism.

Limitations and Future Directions

While the human subject scales and rotates according to camera prompts, the background currently remains static, which can occasionally create a "green screen" effect. Additionally, the system currently uses a lookup table for camera prompts; future iterations could involve automatically regressing camera poses from natural, complex language. As the underlying T2M and T2V sub-modules improve, the overall quality of this framework is expected to scale proportionally.

Summary updated

Abstract

With the rise in expectations related to generative models, text-to-video (T2V) models are being actively studied. Existing text-to-video models have limitations such as in generating complex movements replicating human motions. These model often generate unintended human motions, and the scale of the subject is incorrect. To overcome these limitations and generate high-quality videos that depict human motion under plausible viewing angles, we propose a two stage framework in this study. In the first stage a text-driven human motion generation network generates three-dimensional (3D) human motion from input text prompts and then motion-to-skeleton projection module projects generated motions onto a two-dimensional (2D) skeleton. In the second stage, the projected skeletons are used to generate a video in which the movements of a subject are well-represented. We demonstrated that the proposed framework quantitatively and qualitatively outperforms the existing T2V models. Previously reported human motion generation models use texts only or texts and human skeletons. However, our framework only uses texts and outputs a video related to human motion. Moreover, our framework benefits from using skeleton as an additional condition in the text-to-human motion generation networks. To the best of our knowledge, our framework is the first of its kind that uses text-driven human motion generation networks to generate high-quality videos related to human motions. The corresponding codes are available at https://github.com/CSJasper/HMTV.
BibTeX
@inproceedings{kim2024human,
  author = {Taehoon Kim and Chanhee Kang and JaeHyuk Park and Daun Jeong and ChangHee Yang and Suk-Ju Kang and Kyeongbo Kong},
  title = {{Human Motion Aware Text-to-Video Generation with Explicit Camera Control}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  url = {https://openreview.net/forum?id=xOOedywlfy},
}
Figure