TL;DR
Researchers have developed a two-stage framework that generates high-quality videos of human movement from simple text prompts. By first generating a 3D human motion sequence and then projecting it into 2D skeletons with explicit camera controls (like "top view" or "zoom in"), the system provides a structural "blueprint" that prevents common AI video pitfalls like disappearing limbs, incorrect scaling, and jittery motion.
Addressing the Challenges of Human Motion in T2V
Current text-to-video (T2V) models, while impressive, often struggle when asked to depict complex human actions. They typically suffer from three fundamental issues:
- Motion Ambiguity: The AI might understand a verb like "punching" but generate a static or anatomically impossible pose.
- Scale Problems: Lacking a sense of 3D space, models often produce humans that are disproportionately large or small for their environment.
- Temporal Inconsistency: A subject might look correct in the first frame but morph into a different shape or disappear entirely as the video progresses.
While "skeleton-guided" models exist to mitigate these issues, they usually require users to provide a pre-existing skeleton video. This research introduces a method to generate both the motion and the video from scratch using only text, while granting explicit control over the "virtual camera."
The Two-Stage Framework
The proposed pipeline decouples the structural physics of motion from the stylistic rendering of the video.
Stage 1: Text-to-Skeleton with Camera Control
- Motion Generation: The system processes a text prompt (e.g., "A person is running") using a pre-trained model like T2M-GPT or MDM to generate a sequence of 3D human mesh vertices.
- Explicit Camera Control: This is the core innovation. Users can provide a secondary camera prompt (e.g., "Tilt Down"). The Motion-to-Skeleton Projection Module applies a mathematical projection matrix to the 3D joints. By adjusting rotation and translation parameters based on the camera prompt, it "films" the 3D motion from the desired angle, outputting a 2D skeleton sequence.
Stage 2: Skeleton-guided Video Generation
The 2D skeletons from Stage 1 act as a frame-by-frame structural guide. This sequence, along with the original text prompt, is fed into a diffusion-based T2V model (such as Follow Your Pose). Because the model has a precise map of where the human should be, it can focus its computational resources on rendering textures and backgrounds rather than attempting to learn human anatomy on the fly.
Key Results and Performance
The framework leverages large-scale human motion datasets like HumanML3D and KIT Motion-Language for training. In comparative evaluations, the proposed method significantly outperformed existing T2V models:
- Action Classification (AC): The best configuration (Follow Your Pose + the proposed framework) achieved 48.9% accuracy, compared to 32.5% for ModelScope and 34.6% for Zeroscope.
- Frame Consistency (FC): The model maintained higher visual stability across frames, scoring 92.2% in consistency tests.
- User Preference: In a study with 100 participants, 56.8% of evaluators preferred this framework over competitors like Runway AI or Zeroscope, citing superior semantic relevance and motion realism.
Limitations and Future Directions
While the human subject scales and rotates according to camera prompts, the background currently remains static, which can occasionally create a "green screen" effect. Additionally, the system currently uses a lookup table for camera prompts; future iterations could involve automatically regressing camera poses from natural, complex language. As the underlying T2M and T2V sub-modules improve, the overall quality of this framework is expected to scale proportionally.
Summary updated