Abstract
Artistic style transfer seeks to apply the aesthetic qualities of a style image to a content image while maintaining structural integrity. Despite the success of Large-scale Diffusion Models (LDMs), existing methods often fail to account for the varying levels of abstraction required at different stages of the denoising process and across different neural network layers. We introduce LSAST (Layer-aware and Step-aware Style Transfer), which employs a specialized prompt space that adapts both temporally and architecturally. By utilizing a matrix of 30 learnable prompts and integrating a ControlNet for structural anchoring, LSAST produces highly realistic artistic transfers that outperform current state-of-the-art GAN and diffusion-based models in both visual quality and content preservation.
Introduction
The field of artistic style transfer has transitioned from early optimization-based methods to Generative Adversarial Networks (GANs) and, more recently, Large-scale Diffusion Models (LDMs). While GANs are efficient, they frequently introduce digital artifacts and "disharmonious" patterns. LDMs like Stable Diffusion offer superior realism but face a significant trade-off: aggressive styling often distorts the underlying content, while strict content preservation can result in superficial, filter-like effects.
The primary limitation identified in current diffusion-based style transfer is the use of a "global" prompt. A single text embedding is expected to guide the model through the entire 1,000-step denoising process, ignoring the fact that early steps define coarse layouts while later steps refine fine textures. Similarly, different layers of the U-Net architecture possess different receptive fields and handle different levels of visual abstraction.
Methodology: Dynamic Prompt Spaces
LSAST addresses these limitations through a Step-aware and Layer-aware Prompt Space ($P^*$). This approach decomposes the styling task into two dimensions:
- Temporal Decomposition (Step-aware): The denoising timeline is divided into 10 distinct stages. Each stage is assigned a unique learnable prompt, allowing the model to focus on different stylistic granularities as the image evolves from noise.
- Architectural Decomposition (Layer-aware): The Stable Diffusion U-Net is partitioned into three layers (coarse, moderate, and fine). By assigning specific prompts to these layers, the model can simultaneously manage broad brushstrokes and intricate details.
The resulting 30-prompt matrix is populated via Prompt Inversion. Instead of the computationally prohibitive task of fine-tuning the entire LDM, LSAST only trains these 30 embeddings to reconstruct the style of specific artist collections. To further ensure structural fidelity, a pre-trained ControlNet branch (utilizing Canny edges) is integrated during inference to act as a structural anchor.
Experimental Evaluation
The framework was evaluated using style collections from the AST dataset (including Van Gogh, Monet, and Gauguin) and 10,000 content images from the DIV2K dataset. LSAST was benchmarked against both diffusion-based (InST, ArtBank, ProSpect) and GAN-based (CycleGAN, AST, CUT) methods.
Quantitative Results
LSAST demonstrated superior performance across multiple metrics:
- FID Scores: LSAST consistently achieved the lowest Frechet Inception Distance (FID). For the Van Gogh style, LSAST scored 94.17, a significant improvement over ProSpect (118.22) and CycleGAN (112.04).
- Deception Score: This metric evaluates how often AI-generated art is mistaken for human-created work. LSAST achieved a score of 0.752, nearing the human baseline of 0.863 and outperforming all other models.
Qualitative Comparison and User Preference
Qualitative analysis reveals that LSAST avoids the common artifacts found in GANs and the content distortion typical of other diffusion methods. In a user study involving 5,000 votes, LSAST received a preference score of 0.634, the highest among diffusion-based competitors.
Discussion and Conclusion
LSAST represents a significant advancement in realistic artistic style transfer by acknowledging the spatio-temporal complexity of the diffusion process. While the method offers unparalleled visual quality, it inherits the computational overhead of diffusion models, requiring approximately 4.1 seconds per image compared to the near-instantaneous performance of GANs. Future work will focus on integrating faster sampling techniques, such as Latent Consistency Models (LCMs), to reduce inference time without compromising the stylistic depth achieved by the layer-aware and step-aware prompt space.
Summary updated