Personalized video generation—the ability to place a specific individual into new, AI-generated scenes—represents a significant frontier in generative AI. However, existing methods often face a frustrating trade-off: achieving high identity (ID) fidelity frequently results in "static" videos that lack natural motion, or conversely, maintaining fluid motion leads to a loss of the subject's unique facial features.
A new framework, PersonalVideo, addresses these challenges by moving beyond traditional reconstruction-based tuning. Instead, it employs a reward-based optimization strategy that preserves the dynamic and semantic capabilities of the underlying video model while ensuring precise identity representation.
The Pitfalls of Reconstructive Tuning
Most current customization techniques, such as DreamBooth or MagicMe, rely on reconstructive tuning. This process involves fine-tuning a pre-trained model to reconstruct the specific pixels of a few reference images. While effective for static image generation, this approach introduces two major issues when applied to video:
- Static Bias: Because the model is trained on a small set of static images, its internal "prior" shifts toward stillness. The resulting videos often resemble a slideshow with minimal movement.
- Semantic Degradation: Intensive fine-tuning can "break" the model's ability to follow complex text prompts. The model may overfit to the background or lighting of the reference photo, ignoring the requested actions or environments in the prompt.
The PersonalVideo Framework
PersonalVideo introduces a non-reconstructive approach that optimizes the model based on the quality of the generated video rather than its ability to replicate a static pixel map.
1. Non-Reconstructive Reward Framework
Instead of pixel-matching, PersonalVideo uses differentiable rewards applied to the output of the video generation process:
- Identity Consistency Reward (ICR): Utilizing a pre-trained face recognition model (ArcFace), this reward ensures the generated frames maintain the subject's facial features without forcing the entire image (like the background) to match the reference.
- Semantic Consistency Reward (SCR): This ensures the model retains its original "knowledge" of motion and prompt-following. It compares the semantic distribution of the new model's output to that of the original, frozen Text-to-Video (T2V) model using HPSv2.
2. Isolated Identity Adapter and Prompt Augmentation
The researchers observed that during the diffusion denoising process, motion and layout are established in the early steps, while fine details and identity are filled in later. PersonalVideo exploits this by injecting identity information only during the later denoising stages, preventing identity-specific tuning from interfering with the "physics" of the video.
Furthermore, to prevent overfitting, the framework uses a Large Language Model (LLM) to generate dozens of diverse "simulated prompts" (e.g., "subject surfing" or "subject in a space suit"). Training on these varied scenarios ensures the model learns the person rather than just the specific photo.
Key Results and Performance
PersonalVideo significantly outperforms existing baselines like DreamBooth and MagicMe across both quantitative metrics and human preference:
- Face Similarity: Achieved a score of 62.35, a substantial improvement over MagicMe (50.51) and DreamBooth (42.62).
- Dynamic Degree: Scored 17.80 on the VBench benchmark, proving it maintains motion where other methods tend to produce static results (e.g., MagicMe at 11.88).
- User Preference: In blind studies, PersonalVideo was overwhelmingly preferred for its combination of identity fidelity, text alignment, and overall video quality.
Conclusion and Future Directions
By shifting the focus from pixel reconstruction to reward-based optimization and temporal isolation, PersonalVideo successfully bridges the gap between identity fidelity and motion quality. While currently focused on single-subject customization, the framework lays the groundwork for more complex multi-subject interactions in future generative video research.
Summary updated