Abstract
Artistic style transfer aims to transfer the learned artistic style onto an arbitrary content image, generating artistic stylized images. Existing generative adversarial network-based methods fail to generate highly realistic stylized images and always introduce obvious artifacts and disharmonious patterns. Recently, large-scale pre-trained diffusion models opened up a new way for generating highly realistic artistic stylized images. However, diffusion model-based methods generally fail to preserve the content structure of input content images well, introducing some undesired content structure and style patterns. To address the above problems, we propose a novel pre-trained diffusion-based artistic style transfer method, called LSAST, which can generate highly realistic artistic stylized images while preserving the content structure of input content images well, without bringing obvious artifacts and disharmonious style patterns. Specifically, we introduce a Step-aware and Layer-aware Prompt Space, a set of learnable prompts, which can learn the style information from the collection of artworks and dynamically adjusts the input images' content structure and style pattern. To train our prompt space, we propose a novel inversion method, called Step-ware and Layer-aware Prompt Inversion, which allows the prompt space to learn the style information of the artworks collection. In addition, we inject a pre-trained conditional branch of ControlNet into our LSAST, which further improved our framework's ability to maintain content structure. Extensive experiments demonstrate that our proposed method can generate more highly realistic artistic stylized images than the state-of-the-art artistic style transfer methods.
Introduction
Artistic Style Transfer (AST) is the process of migrating the aesthetic characteristics of an artwork—such as its color palette, brushwork, and texture—onto a target content image. Historically, this field has been dominated by Generative Adversarial Networks (GANs). While GANs are efficient, they often struggle with realism, frequently producing "disharmonious" patterns or visible artifacts that break the illusion of a cohesive painting.
With the advent of large-scale diffusion models like Stable Diffusion, the potential for high-fidelity artistic generation has increased significantly. However, these models introduce a new challenge: the "realism vs. structure" trade-off. While diffusion models excel at creating rich textures, they often "hallucinate" or warp the underlying structure of the content image, turning recognizable objects into abstract shapes. This paper introduces LSAST (Layer-aware and Step-aware Artistic Style Transfer), a framework designed to bridge this gap by providing fine-grained control over the stylization process.
The LSAST Framework
The core innovation of LSAST lies in the recognition that style is not a monolithic attribute. It manifests differently across different scales (coarse vs. fine) and at different stages of the image generation process. To capture this nuance, the authors propose three primary technical components.
1. Step-aware and Layer-aware Prompt Space ($P^*$)
Traditional diffusion-based style transfer often relies on a single global text embedding to represent a style. LSAST replaces this with a set of 30 learnable prompts. The framework divides the 1,000-step denoising process into 10 distinct stages and the U-Net architecture into 3 layer groups (coarse, moderate, and fine). This allows the model to apply specific "stylistic instructions" depending on whether it is currently establishing the broad composition of the image or refining the minute details of the brushstrokes.
2. Step-aware and Layer-aware Prompt Inversion
To populate this prompt space, the authors developed a novel inversion method. Rather than learning from a single reference image—which can lead to overfitting on a specific composition—the model learns from a collection of an artist's work (e.g., a dataset of Van Gogh paintings). This allows the prompt space to capture the general "essence" of the artist's style, which can then be applied more flexibly to new content.
3. Structural Preservation via ControlNet
To ensure the final output remains faithful to the original content's layout, LSAST integrates a pre-trained ControlNet branch. By utilizing Canny edge detection during the inference stage, the model uses the edges of the original image as a structural "anchor." This prevents the diffusion process from moving objects or altering the fundamental geometry of the scene, a common failure point in previous diffusion-based methods like InST or ArtBank.
Experimental Results
The researchers evaluated LSAST against state-of-the-art (SOTA) methods, including CycleGAN and various diffusion-based competitors, using the AST dataset for styles and DIV2K for content images.
Quantitative Performance
LSAST demonstrated superior performance across several metrics, most notably the Fréchet Inception Distance (FID), which measures the statistical similarity between generated images and real artworks. In the Van Gogh style category, LSAST achieved an FID of 94.17, significantly outperforming the next best diffusion method, ProSpect, which scored 118.22.
Furthermore, the model achieved high Deception Scores in user studies. When asked to distinguish between AI-generated art and real human-made paintings, participants gave LSAST a score of 0.752. For context, ground-truth human art received a score of 0.863, while GAN-based methods typically hovered around 0.5.
Qualitative Observations
Visually, LSAST produces images that maintain the integrity of the content while applying sophisticated artistic textures. Unlike earlier models that might turn a house into a vague blob of color, LSAST preserves the sharp edges of the architecture while imbuing the surfaces with the characteristic thick, rhythmic brushstrokes of the target style.
Conclusion and Future Work
LSAST represents a significant step forward in making diffusion-based style transfer both realistic and structurally reliable. By decomposing style into temporal (step-aware) and architectural (layer-aware) components, the framework achieves a level of nuance previously unavailable in AST.
However, this quality comes at a computational cost. LSAST requires approximately 4.13 seconds to generate an image, compared to the near-instantaneous 0.03 seconds of GAN-based methods. Future research will likely focus on optimizing diffusion sampling techniques to reduce this latency and exploring the application of step-and-layer logic to video style transfer, where temporal consistency is paramount.
Summary updated