PersonalVideo: High ID-Fidelity Video Customization without Dynamic and Semantic Degradation

Hengjia Li, Haonan Qiu, Shiwei Zhang, Xiang Wang, Yujie Wei, Zekun Li, Yingya Zhang, Boxi Wu, Deng Cai

Personalized video generation—the ability to place a specific individual into new, AI-generated scenes—represents a significant frontier in generative AI. However, existing methods often face a frustrating trade-off: achieving high identity (ID) fidelity frequently results in "static" videos that lack natural motion, or conversely, maintaining fluid motion leads to a loss of the subject's unique facial features.

A new framework, PersonalVideo, addresses these challenges by moving beyond traditional reconstruction-based tuning. Instead, it employs a reward-based optimization strategy that preserves the dynamic and semantic capabilities of the underlying video model while ensuring precise identity representation.

The Pitfalls of Reconstructive Tuning

Most current customization techniques, such as DreamBooth or MagicMe, rely on reconstructive tuning. This process involves fine-tuning a pre-trained model to reconstruct the specific pixels of a few reference images. While effective for static image generation, this approach introduces two major issues when applied to video:

  1. Static Bias: Because the model is trained on a small set of static images, its internal "prior" shifts toward stillness. The resulting videos often resemble a slideshow with minimal movement.
  2. Semantic Degradation: Intensive fine-tuning can "break" the model's ability to follow complex text prompts. The model may overfit to the background or lighting of the reference photo, ignoring the requested actions or environments in the prompt.

The PersonalVideo Framework

PersonalVideo introduces a non-reconstructive approach that optimizes the model based on the quality of the generated video rather than its ability to replicate a static pixel map.

1. Non-Reconstructive Reward Framework

Instead of pixel-matching, PersonalVideo uses differentiable rewards applied to the output of the video generation process:

  • Identity Consistency Reward (ICR): Utilizing a pre-trained face recognition model (ArcFace), this reward ensures the generated frames maintain the subject's facial features without forcing the entire image (like the background) to match the reference.
  • Semantic Consistency Reward (SCR): This ensures the model retains its original "knowledge" of motion and prompt-following. It compares the semantic distribution of the new model's output to that of the original, frozen Text-to-Video (T2V) model using HPSv2.
PersonalVideo maintains high identity fidelity while allowing for complex, natural motions like playing a guitar.

2. Isolated Identity Adapter and Prompt Augmentation

The researchers observed that during the diffusion denoising process, motion and layout are established in the early steps, while fine details and identity are filled in later. PersonalVideo exploits this by injecting identity information only during the later denoising stages, preventing identity-specific tuning from interfering with the "physics" of the video.

Furthermore, to prevent overfitting, the framework uses a Large Language Model (LLM) to generate dozens of diverse "simulated prompts" (e.g., "subject surfing" or "subject in a space suit"). Training on these varied scenarios ensures the model learns the person rather than just the specific photo.

The framework preserves the semantic "magic" of the base model, enabling high-quality visual effects and complex scene interactions.

Key Results and Performance

PersonalVideo significantly outperforms existing baselines like DreamBooth and MagicMe across both quantitative metrics and human preference:

  • Face Similarity: Achieved a score of 62.35, a substantial improvement over MagicMe (50.51) and DreamBooth (42.62).
  • Dynamic Degree: Scored 17.80 on the VBench benchmark, proving it maintains motion where other methods tend to produce static results (e.g., MagicMe at 11.88).
  • User Preference: In blind studies, PersonalVideo was overwhelmingly preferred for its combination of identity fidelity, text alignment, and overall video quality.

Conclusion and Future Directions

By shifting the focus from pixel reconstruction to reward-based optimization and temporal isolation, PersonalVideo successfully bridges the gap between identity fidelity and motion quality. While currently focused on single-subject customization, the framework lays the groundwork for more complex multi-subject interactions in future generative video research.

Summary updated

Abstract

The current text-to-video (T2V) generation has made significant progress in synthesizing realistic general videos, but it is still under-explored in identity-specific human video generation with customized ID images. The key challenge lies in maintaining high ID fidelity consistently while preserving the original motion dynamic and semantic following after the identity injection. Current video identity customization methods mainly rely on reconstructing given identity images on text-to-image models, which have a divergent distribution with the T2V model. This process introduces a tuning-inference gap, leading to dynamic and semantic degradation. To tackle this problem, we propose a novel framework, dubbed $\textbf{PersonalVideo}$, that applies a mixture of reward supervision on synthesized videos instead of the simple reconstruction objective on images. Specifically, we first incorporate identity consistency reward to effectively inject the reference's identity without the tuning-inference gap. Then we propose a novel semantic consistency reward to align the semantic distribution of the generated videos with the original T2V model, which preserves its dynamic and semantic following capability during the identity injection. With the non-reconstructive reward training, we further employ simulated prompt augmentation to reduce overfitting by supervising generated results in more semantic scenarios, gaining good robustness even with only a single reference image. Extensive experiments demonstrate our method's superiority in delivering high identity faithfulness while preserving the inherent video generation qualities of the original T2V model, outshining prior methods.
BibTeX
@inproceedings{li2025personalvideo,
  author = {Hengjia Li and Haonan Qiu and Shiwei Zhang and Xiang Wang and Yujie Wei and Zekun Li and Yingya Zhang and Boxi Wu and Deng Cai},
  title = {{PersonalVideo: High ID-Fidelity Video Customization without Dynamic and Semantic Degradation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure