TL;DR
This paper introduces a method to personalize pose-driven human video diffusion models using only a single reference image, eliminating the need for subject-specific videos or manual labels. By synthesizing high-quality "pseudo-ground-truth" frames and using them for test-time adaptation via static clips, the approach significantly improves identity preservation and video quality simultaneously.
What Problem Does This Paper Address?
Generating a video of a specific person from a single image—a task known as pose-driven human animation—is notoriously difficult. While modern video diffusion models can produce realistic motion, they often suffer from identity drift. As the subject moves or turns away from the original reference pose, their facial features, clothing details, and logos often distort or vanish.
The standard fix is "personalization" (fine-tuning the model on the specific subject). However, effective personalization usually requires a video or multiple images of that person to teach the model what they look like from different angles. In many real-world scenarios, we only have one image. This paper tackles the challenge of personalizing a model when one image is all you have.
Methods
The authors propose a two-stage framework that generates its own supervision at test time:
-
Reward-Guided Pseudo-Frame Generation: The system first selects a few "key poses" from the target motion sequence. It uses a domain-tuned, pose-conditioned image generator (based on FLUX.1) to create synthetic images of the subject in these poses. To ensure these synthetic frames are accurate, the authors use an inference-time "particle-steering" procedure. Multiple candidate images are generated and "steered" toward the subject's identity using rewards from DINO (for body similarity) and ArcFace (for facial identity).
-
Static-Clip Adaptation: Once high-quality "pseudo-ground-truth" frames are generated, the system fine-tunes a lightweight LoRA (Low-Rank Adaptation) on the base video model. A key innovation here is the use of static clips: each synthetic frame is replicated into a short video with zero motion. Training on these static clips allows the model to learn the subject's appearance across different poses without damaging its internal "motion prior"—the knowledge of how humans naturally move.
Datasets
The model was evaluated on:
- TikTok Dataset: A standard benchmark for human dance animation.
- UBC Fashion: A dataset of people walking in various outfits.
- Unseen100-Val: A new, more challenging benchmark introduced by the authors. It consists of 100 in-the-wild videos featuring diverse subjects (including 2D/3D animated characters), complex lighting, and varied backgrounds to test real-world generalization.
Key Insights and Ideas
- Synthetic is Better than Real: Surprisingly, the authors found that synthetic frames can be more effective for supervision than real video frames. Real frames often contain motion blur or temporal artifacts that can "poison" the model's motion layers. Clean, static synthetic frames provide a "pure" appearance signal.
- The Identity-vs-Motion Trade-off: Usually, personalizing a model improves identity but makes the video look "jittery" or static (high FVD). By using the static-clip training strategy, this method actually improves video quality (lowers FVD) while boosting identity preservation.
- Farthest-Point Sampling: Instead of picking random frames for supervision, the method uses farthest-point sampling in pose space. This ensures the model is trained on the most "difficult" poses—those furthest from the original reference image.
Key Results
The method, dubbed UA-DiT, outperforms state-of-the-art baselines across multiple metrics:
- Video Quality: On the TikTok benchmark, it achieved an FVD of 133.5, a massive improvement over the non-personalized base model (402.1) and significantly better than competitors like MusePose (532.7) or Moore-AnimateAnyone (501.2).
- Identity Preservation: In cross-reenactment tests (where one person's motion drives another's appearance), it boosted FaceID scores by +0.140 over the base model.
- Temporal Stability: The adapted model reduces mean inter-frame residual (flicker) by 60%, leading to much smoother videos.
Limitations and Future Work
- Generator Dependence: The quality of the final video is bounded by the quality of the initial image generator. If the generator produces a blurry or distorted pseudo-frame, the adaptation will suffer.
- Test-Time Compute: While inference is fast, the one-time personalization process takes about 9.3 minutes on an H100 GPU. This makes it more suitable for "enrolling" a character once for repeated use rather than instant, one-off generations.
- Future Directions: The authors suggest a "closed-loop" approach where the adapted model's own high-quality outputs could be used to further refine the initial generator.
Summary updated