Introduction
Face stylization—the process of transforming a photograph into an artistic portrait—has become a cornerstone of creative AI. However, achieving high-quality results with only a single target style image (one-shot) remains a significant challenge. Traditional Generative Adversarial Network (GAN) based methods often struggle with geometric rigidity, failing to replicate exaggerated artistic features like the large eyes common in anime, and frequently suffer from content entanglement, where source details like hair color or pose are lost.
The StyO Framework
To overcome these limitations, the authors propose StyO, a framework built upon Latent Diffusion Models (LDMs). StyO employs a "disentangle-and-recombine" strategy to separate facial content from artistic style effectively.
Identifier Disentanglement Learner (IDL)
The IDL component uses unique text identifiers (tokens) to represent content and style separately. By employing contrastive prompting—such as "a drawing with [Positive Style][Negative Style] style of [Positive Content][Negative Content] portrait"—and a triple reconstruction loss, the model learns to isolate artistic elements from human facial features. This prevents the style of the target from bleeding into the content representation of the source.
Fine-grained Content Controller (FCC)
To ensure the source's pose and features are preserved, the FCC manipulates the cross-attention layers of the LDM. It extracts spatial attention maps from the source image and injects them into the stylization process, acting as a spatial anchor. Additionally, prompt augmentation allows users to adjust the intensity of the stylization by repeating style identifiers.
Key Results and Comparison
StyO demonstrates superior performance compared to state-of-the-art models like JoJoGAN. Unlike GAN-based approaches, StyO's use of diffusion allows for greater flexibility with "out-of-distribution" shapes, such as caricatures or stylized paintings, while maintaining the source's identity.
In qualitative evaluations, StyO consistently matches target brushstrokes and geometry while retaining source characteristics like hair color and head orientation. Quantitative user studies further validate these findings, with StyO being preferred for identity preservation (42%), geometric accuracy (47%), and texture quality (38%).
Conclusion
StyO represents a significant advancement in one-shot face stylization. By leveraging the flexibility of Latent Diffusion Models and introducing novel disentanglement and control mechanisms, it achieves a balance between artistic expression and identity preservation that previous GAN-based methods could not reach. Future work will explore automated weight balancing and extending these capabilities to video stylization.
Summary updated