Introduction
Stylizing a single image has become a relatively mature task, but extending this to video remains a significant challenge. Applying artistic styles (such as 3D animation or sketches) to video frames individually often results in three primary issues: temporal inconsistency (flickering), loss of original facial expressions, and rigid alignment constraints that fail during dynamic movement. While personalized diffusion models like DreamBooth offer high-quality static results, they lack the inherent frame-to-frame dependencies required for stable video synthesis.
Proposed Method: Local Consistency Guidance (LCG)
To address these challenges, we propose the Local Consistency Guidance (LCG) framework. This method enhances a Personalized Diffusion Model (leveraging DreamBooth and ControlNet) with two key temporal pillars designed to enforce stability at the latent level.
Local Cross-Frame Attention
Standard U-Net architectures utilize self-attention within individual frames. LCG replaces these with a local cross-attention mechanism. By employing a sliding window that observes neighboring frames and applying weights that decay with temporal distance, the model ensures the current frame remains mathematically consistent with its immediate neighbors.
Inter-framed Local Style Transfer
To preserve fine-grained details, we integrate an example-based style transfer method directly into the iterative denoising process (DDIM). Rather than relying solely on the diffusion model's generative capabilities, LCG uses stylized information from the previous frame to guide the denoising of the current frame at each step. This hybrid approach resolves potential contradictions between frames before they manifest as visual artifacts.
Experimental Results
The framework was evaluated using the VFHQ (Video Face High Quality) dataset and compared against baselines such as EbSynth and JOJOGan.
Quantitative Performance
Using a "Frame Consistency" metric, the LCG method demonstrated superior stability:
- Source Video: 0.91 (Baseline stability)
- DreamBooth + ControlNet: 0.83 (Standard diffusion decreased stability)
- LCG (Ours): 0.97
Remarkably, the LCG method improved upon the temporal consistency of the original source footage, providing a smoother visual experience than the raw input.
Qualitative Analysis
Qualitatively, the method excels at preserving the nuance of the actor's original expressions while adhering strictly to the personalized style. As shown in the Elsa-styled output below, the model maintains high-fidelity facial features and makeup that traditional GAN-based methods often struggle to replicate.
Conclusion and Future Work
Local Consistency Guidance successfully bridges the gap between personalized image generation and stable video stylization by injecting consistency during the denoising phase. Future research will focus on expanding the diversity of artistic styles, optimizing for more complex facial environments, and reducing the computational overhead associated with iterative local style transfer.
Summary updated