Magic-Me: Identity-Specific Video Customized Diffusion
Introduction
Generating high-quality videos of specific individuals performing arbitrary actions in diverse settings is a significant challenge in generative AI. While text-to-image models have achieved remarkable success in subject-driven customization using techniques like LoRA or DreamBooth, extending these capabilities to video has been hindered by issues such as "identity drift"—where the subject's appearance changes between frames—and temporal instability. Researchers from ByteDance and UC Berkeley have introduced Magic-Me, a novel framework for Video Custom Diffusion (VCD) designed to produce identity-consistent videos from a limited set of reference images.
Challenges in Video Customization
Subject-driven video generation faces two primary technical hurdles:
- Inter-frame Instability: Maintaining a consistent pose and appearance across frames is difficult because standard models often initialize each frame with independent noise, leading to flickering and erratic motion.
- Identity Preservation in Full-Body Shots: In many video scenarios, the face occupies a small portion of the frame. Standard Variational Autoencoder (VAE) decoders often lose the fine-grained details necessary for identity recognition at these resolutions, resulting in generic-looking subjects.
The Magic-Me Framework
Magic-Me addresses these challenges through a modular three-stage pipeline and innovative noise priors.
3D Gaussian Noise Prior
To improve temporal stability without extensive retraining, VCD employs a 3D Gaussian noise prior. By using a covariance matrix to correlate initial noise across the temporal dimension, the model ensures that the starting point for each frame is mathematically linked. This training-free method significantly reduces motion jitter and artifacts.
Extended Textual Inversion (TI) with Segmentation
To learn a specific identity, Magic-Me enhances standard Textual Inversion by using multiple learnable tokens and a "prompt-to-segmentation" module (utilizing Grounding DINO and SAM). This allows the model to focus exclusively on the subject's features while ignoring background noise in the reference images.
The Three-Stage VCD Pipeline
The generation process is divided into three distinct phases:
- T2V VCD: Generates a base, low-resolution video aligned with the text prompt.
- Face VCD: Automatically detects and upscales the facial region, then applies "partial denoising" to inject high-fidelity identity details. This stage is crucial for overcoming the resolution limits of standard latent diffusion.
- Tiled VCD: Upscales the entire video to 1024x1024 using a tiled approach to maintain high resolution across the frame without exceeding memory constraints.
Results and Evaluation
The framework was evaluated using the CustomConcept101 dataset and web-collected images, with prompts generated by GPT-4V to create a comprehensive benchmark of 2,400 videos. Magic-Me demonstrated superior performance over baselines like IP-Adapter and DreamBooth across several key metrics:
- Identity Similarity (DINO): Achieved a score of 0.585.
- Text Alignment (CLIP-T): Scored 0.315, indicating strong adherence to complex prompts.
- Temporal Consistency (CLIP-I Temp): Reached 0.835, reflecting high frame-to-frame stability.
Conclusion and Future Work
Magic-Me represents a significant step forward in customized video generation, offering a "plug-and-play" framework compatible with community models and ControlNet. While it excels at single-subject consistency, future work will focus on multi-identity interactions and extending the duration of generated clips beyond the current 16-frame limit imposed by underlying motion modules.
Summary updated