Abstract
Artistic style transfer aims to transfer the learned artistic style onto an arbitrary content image, generating artistic stylized images. Existing generative adversarial network-based methods fail to generate highly realistic stylized images and always introduce obvious artifacts and disharmonious patterns. Recently, large-scale pre-trained diffusion models opened up a new way for generating highly realistic artistic stylized images. However, diffusion model-based methods generally fail to preserve the content structure of input content images well, introducing some undesired content structure and style patterns. To address the above problems, we propose a novel pre-trained diffusion-based artistic style transfer method, called LSAST, which can generate highly realistic artistic stylized images while preserving the content structure of input content images well, without bringing obvious artifacts and disharmonious style patterns. Specifically, we introduce a Step-aware and Layer-aware Prompt Space, a set of learnable prompts, which can learn the style information from the collection of artworks and dynamically adjusts the input images' content structure and style pattern. To train our prompt space, we propose a novel inversion method, called Step-ware and Layer-aware Prompt Inversion, which allows the prompt space to learn the style information of the artworks collection. In addition, we inject a pre-trained conditional branch of ControlNet into our LSAST, which further improved our framework's ability to maintain content structure. Extensive experiments demonstrate that our proposed method can generate more highly realistic artistic stylized images than the state-of-the-art artistic style transfer methods.
Introduction
Artistic Style Transfer (AST) is the process of synthesizing a new image by combining the semantic content of a photograph with the stylistic characteristics of an artwork. Traditionally, Generative Adversarial Networks (GANs) have been the dominant architecture for this task. While efficient, GANs often produce "disharmonious" patterns or messy artifacts that detract from the realism of the generated art.
The advent of large-scale pre-trained diffusion models, such as Stable Diffusion, has shifted the landscape. These models excel at generating high-quality textures but often struggle with content preservation. During the iterative denoising process, diffusion models frequently "hallucinate" new structures, causing the original photo's composition to warp. The authors of LSAST address this by introducing a Step-aware and Layer-aware Prompt Space, arguing that a single, global text prompt is insufficient to guide the complex, multi-stage process of image synthesis.
Methodology: Step-aware and Layer-aware Prompting
The core innovation of LSAST is the Step-aware and Layer-aware Prompt Space ($P^*$). Instead of a static embedding, the model uses a dynamic set of learnable prompts to control the stylization process across both time (denoising steps) and space (network layers).
1. Step-aware and Layer-aware Control
The standard 1,000-step denoising process is divided into 10 discrete stages. Since early steps typically define coarse structure and later steps define fine texture, each stage is assigned a unique, learnable prompt. Furthermore, the U-Net architecture is divided into three layers—coarse, moderate, and fine—each receiving specific prompts to control style information at different spatial resolutions.
2. Prompt Inversion and Structural Guidance
To train these prompts, the researchers employ Step-aware and Layer-aware Prompt Inversion. By freezing the pre-trained Stable Diffusion model and optimizing only the prompt space, the model learns the essence of an artist’s collection (e.g., Van Gogh or Monet) without the risk of "catastrophic forgetting." To ensure structural rigidity, a pre-trained ControlNet branch is integrated as a "content prompt," locking the "skeleton" of the input image in place during stylization.
Experimental Results
LSAST was evaluated against state-of-the-art (SOTA) diffusion methods (InST, ArtBank, ProSpect) and GAN-based methods (CycleGAN, AST, LSeSim, CUT) using style collections from artists like Van Gogh and Monet.
Qualitative Comparison
Visual results demonstrate that LSAST generates images with superior realism and structural fidelity. While other diffusion methods often introduce undesired artifacts or lose the original content's composition, LSAST maintains the integrity of the input while applying sophisticated artistic textures.
Quantitative Metrics
- FID (Frechet Inception Distance): LSAST achieved the lowest FID scores in nearly all categories (e.g., 94.17 for Van Gogh), indicating a distribution closest to real artwork.
- Deception Score: In human evaluation, LSAST achieved a score of 0.752, significantly outperforming GAN-based methods (~0.58) and approaching the score of real art (0.863).
- User Preference: In A/B testing, users preferred LSAST results 63.4% of the time compared to other SOTA methods.
Conclusion and Limitations
LSAST represents a significant step forward in realistic artistic style transfer by leveraging the power of diffusion models with fine-grained control. However, the iterative nature of diffusion makes it slower than GANs, with an inference time of approximately 4.1 seconds per image compared to 0.03 seconds for CycleGAN. Future work will focus on accelerating the sampling process and exploring how to automatically generalize the prompt space to unseen styles.
Summary updated