StyO: Stylize Your Face in Only One-Shot

Bonan Li, Zicheng Zhang, Xuecheng Nie, Congying Han, Yinhan Hu, Xinmin Qiu, Tiande Guo

Introduction

Face stylization—the process of transforming a photograph into an artistic portrait—has become a cornerstone of creative AI. However, achieving high-quality results with only a single target style image (one-shot) remains a significant challenge. Traditional Generative Adversarial Network (GAN) based methods often struggle with geometric rigidity, failing to replicate exaggerated artistic features like the large eyes common in anime, and frequently suffer from content entanglement, where source details like hair color or pose are lost.

The StyO Framework

To overcome these limitations, the authors propose StyO, a framework built upon Latent Diffusion Models (LDMs). StyO employs a "disentangle-and-recombine" strategy to separate facial content from artistic style effectively.

Identifier Disentanglement Learner (IDL)

The IDL component uses unique text identifiers (tokens) to represent content and style separately. By employing contrastive prompting—such as "a drawing with [Positive Style][Negative Style] style of [Positive Content][Negative Content] portrait"—and a triple reconstruction loss, the model learns to isolate artistic elements from human facial features. This prevents the style of the target from bleeding into the content representation of the source.

Fine-grained Content Controller (FCC)

To ensure the source's pose and features are preserved, the FCC manipulates the cross-attention layers of the LDM. It extracts spatial attention maps from the source image and injects them into the stylization process, acting as a spatial anchor. Additionally, prompt augmentation allows users to adjust the intensity of the stylization by repeating style identifiers.

Key Results and Comparison

StyO demonstrates superior performance compared to state-of-the-art models like JoJoGAN. Unlike GAN-based approaches, StyO's use of diffusion allows for greater flexibility with "out-of-distribution" shapes, such as caricatures or stylized paintings, while maintaining the source's identity.

Comparison of StyO (Ours) against JoJoGAN across various source faces and target styles. StyO better preserves identity while capturing complex geometric styles.

In qualitative evaluations, StyO consistently matches target brushstrokes and geometry while retaining source characteristics like hair color and head orientation. Quantitative user studies further validate these findings, with StyO being preferred for identity preservation (42%), geometric accuracy (47%), and texture quality (38%).

Conclusion

StyO represents a significant advancement in one-shot face stylization. By leveraging the flexibility of Latent Diffusion Models and introducing novel disentanglement and control mechanisms, it achieves a balance between artistic expression and identity preservation that previous GAN-based methods could not reach. Future work will explore automated weight balancing and extending these capabilities to video stylization.

Summary updated

Abstract

This paper focuses on face stylization with a single artistic target. Existing works for this task often fail to retain the source content while achieving geometry variation. Here, we present a novel StyO model, i.e., Stylize the face in only One-shot, to solve the above problem. In particular, StyO exploits a disentanglement and recombination strategy. It first disentangles the content and style of source and target images into identifiers, which are then recombined in a cross manner to derive the stylized face image. In this way, StyO decomposes complex images into independent and specific attributes, and simplifies one-shot face stylization as the combination of different attributes from input images, thus producing results better matching face geometry of target image and content of source one. StyO is implemented with latent diffusion models (LDM) and composed of two key modules: 1) Identifier Disentanglement Learner (IDL) for disentanglement phase. It represents identifiers as contrastive text prompts, i.e. positive and negative descriptions. And it introduces a novel triple reconstruction loss to fine-tune the pre-trained LDM for encoding style and content into corresponding identifiers; 2) Fine-graind Content Controller (FCC) for recombination phase. It recombines disentangled identifiers from IDL to form an augmented text prompt for generating stylized faces. In addition, FCC also constrains the cross-attention maps of latent and text features to preserve source face details in results. The extensive evaluation shows that StyO produces high-quality images on numerous paintings of various styles and outperforms the current state-of-the-art.
BibTeX
@inproceedings{li2025styo,
  author = {Bonan Li and Zicheng Zhang and Xuecheng Nie and Congying Han and Yinhan Hu and Xinmin Qiu and Tiande Guo},
  title = {{StyO: Stylize Your Face in Only One-Shot}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i5.32488},
}
Figure