Local Consistency Guidance: Personalized Stylization Method of Face Video (Student Abstract)

Wancheng Feng, Yingchao Liu, Jiaming Pei, Wenxuan Liu, Chunpeng Tian, Lukun WangAAAI 2024 · DOI

Introduction

Stylizing a single image has become a relatively mature task, but extending this to video remains a significant challenge. Applying artistic styles (such as 3D animation or sketches) to video frames individually often results in three primary issues: temporal inconsistency (flickering), loss of original facial expressions, and rigid alignment constraints that fail during dynamic movement. While personalized diffusion models like DreamBooth offer high-quality static results, they lack the inherent frame-to-frame dependencies required for stable video synthesis.

Proposed Method: Local Consistency Guidance (LCG)

To address these challenges, we propose the Local Consistency Guidance (LCG) framework. This method enhances a Personalized Diffusion Model (leveraging DreamBooth and ControlNet) with two key temporal pillars designed to enforce stability at the latent level.

Flowchart of the LCG framework and the structure of Local Style Transfer.

Local Cross-Frame Attention

Standard U-Net architectures utilize self-attention within individual frames. LCG replaces these with a local cross-attention mechanism. By employing a sliding window that observes neighboring frames and applying weights that decay with temporal distance, the model ensures the current frame remains mathematically consistent with its immediate neighbors.

Inter-framed Local Style Transfer

To preserve fine-grained details, we integrate an example-based style transfer method directly into the iterative denoising process (DDIM). Rather than relying solely on the diffusion model's generative capabilities, LCG uses stylized information from the previous frame to guide the denoising of the current frame at each step. This hybrid approach resolves potential contradictions between frames before they manifest as visual artifacts.

Experimental Results

The framework was evaluated using the VFHQ (Video Face High Quality) dataset and compared against baselines such as EbSynth and JOJOGan.

Quantitative Performance

Using a "Frame Consistency" metric, the LCG method demonstrated superior stability:

  • Source Video: 0.91 (Baseline stability)
  • DreamBooth + ControlNet: 0.83 (Standard diffusion decreased stability)
  • LCG (Ours): 0.97

Remarkably, the LCG method improved upon the temporal consistency of the original source footage, providing a smoother visual experience than the raw input.

Qualitative Analysis

Qualitatively, the method excels at preserving the nuance of the actor's original expressions while adhering strictly to the personalized style. As shown in the Elsa-styled output below, the model maintains high-fidelity facial features and makeup that traditional GAN-based methods often struggle to replicate.

Example of high-quality personalized face stylization (Elsa style).

Conclusion and Future Work

Local Consistency Guidance successfully bridges the gap between personalized image generation and stable video stylization by injecting consistency during the denoising phase. Future research will focus on expanding the diversity of artistic styles, optimizing for more complex facial environments, and reducing the computational overhead associated with iterative local style transfer.

Summary updated

BibTeX
@inproceedings{feng2024local,
  author = {Wancheng Feng and Yingchao Liu and Jiaming Pei and Wenxuan Liu and Chunpeng Tian and Lukun Wang},
  title = {{Local Consistency Guidance: Personalized Stylization Method of Face Video (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaai.v38i21.30440},
}
Figure