A Comprehensive Survey on Human Video Generation: Challenges, Methods, and Insights

Wentao Lei, Jinting Wang, Fengji Ma, Guanjie Huang, Li LiuarXiv 2024 · DOI · arXiv

Introduction

The field of human video generation has seen a rapid transformation, moving from simple facial animations to complex full-body synthesis. Generating realistic videos of humans presents unique challenges compared to general video generation. Humans are highly sensitive to the "uncanny valley," where even minor anatomical errors—such as "spaghetti fingers" or unnatural joint rotations—immediately break the illusion of realism. Furthermore, maintaining temporal consistency (ensuring a person's appearance doesn't flicker or change between frames) and achieving perfect alignment with driving signals like speech or music remain significant hurdles. This survey provides a systematic overview of the current landscape, categorizing methods by their driving modalities and identifying the critical bottlenecks that persist in the quest for photorealism.

Methodological Pillars

Research in human video generation is primarily organized around three "control signals" that drive the synthesis process. These modalities allow for varying degrees of control over the resulting video.

Overview of driving modalities for human video generation, including audio, pose, and text-based signals.

1. Text-to-Human

Text-driven methods allow users to specify appearance (e.g., clothing style) or motion (e.g., "a person walking"). Most state-of-the-art approaches utilize a two-stage pipeline: first, the text prompt is mapped to a sequence of intermediate poses or action descriptions, which are then rendered into the final video frames.

2. Audio-to-Human

This modality focuses on speech-driven gestures (co-speech) and music-driven dance. The primary technical challenge is "Audio2motion," which requires the model to capture the rhythm, prosody, and emotional nuances of the audio and translate them into naturalistic body language.

3. Pose-to-Human

Pose-based control offers the most direct manipulation of human movement. Researchers employ various structural "priors" to guide generative models, ranging from 2D skeletons to dense surface mappings and 3D meshes. These representations act as structural anchors, preventing the human body from deforming into unrealistic shapes during the generation process.

A 2D skeleton pose representation used as an intermediate structural anchor for human motion synthesis.

Technological Evolution: From GANs to Diffusion

For years, Generative Adversarial Networks (GANs) were the standard for human synthesis. However, the field has undergone a "Diffusion Revolution." Diffusion Models, such as Stable Diffusion and Stable Video Diffusion (SVD), now dominate due to their superior visual quality and diversity. While GANs are faster for inference, Diffusion Models have significantly reduced the blurriness and artifacts common in earlier work. Current research is increasingly exploring Diffusion Transformers (DiT) to better handle the long-range temporal dependencies required for complex video sequences.

Datasets and Benchmarks

The survey identifies a wide array of task-specific datasets:

  • Action & Motion: UCF-101, Human3.6m, and NTU RGB+D provide foundational data for general movement.
  • Dance & Performance: AIST++ (music-aligned) and TikTok datasets are crucial for rhythmic synthesis, while URMP focuses on musical instrument performance.
  • Fashion & Sign Language: DeepFashion and MS-ASL/PHOENIX14T enable specialized synthesis for virtual try-ons and accessibility tools.

Key Insights and Challenges

A central challenge in the field is the "One-to-Many" mapping problem: a single prompt or audio clip can result in infinite valid motions, making it difficult to define a single "ground truth" for training. To address this, successful models often focus on disentanglement—separating the modeling of identity (appearance) from motion to ensure consistency across frames.

Despite recent progress, six major hurdles remain:

  1. Occlusion: Handling limbs that disappear and reappear behind the body.
  2. Body Deformation: Preventing anatomical glitches like malformed hands.
  3. Appearance Inconsistency: Eliminating flickering in clothing or facial features.
  4. Background Influence: Preventing the subject from "bleeding" into or warping the background.
  5. Temporal Inalignment: Ensuring the video perfectly matches the timing of the driving signal.
  6. Unnatural Poses: Moving beyond mathematically possible poses to those that look physically natural.

Future Directions

Future research must address the lack of "GPT-level" massive datasets for human video, which are currently small and task-specific. Additionally, extending the duration of generated videos from seconds to minutes and enabling fine-grained, interactive control (such as adjusting a specific finger's movement) remain key frontiers for the community.

Summary updated

BibTeX
@misc{lei2024comprehensive,
  author = {Wentao Lei and Jinting Wang and Fengji Ma and Guanjie Huang and Li Liu},
  title = {{A Comprehensive Survey on Human Video Generation: Challenges, Methods, and Insights}},
  year = {2024},
  eprint = {2407.08428},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.08428},
}
Figure