SPAST: Arbitrary Style Transfer with Style Priors via Pre-trained Large-scale Model

Zhanjie Zhang, Quanwei Zhang, Junsheng Luan, Mengyuan Yang, Yun Wang, Lei Zhao

Abstract

Given an arbitrary content and style image, arbitrary style transfer aims to render a new stylized image which preserves the content image's structure and possesses the style image's style. Existing arbitrary style transfer methods are based on either small models or pre-trained large-scale models. The small model-based methods fail to generate high-quality stylized images, bringing artifacts and disharmonious patterns. The pre-trained large-scale model-based methods can generate high-quality stylized images but struggle to preserve the content structure and cost long inference time. To this end, we propose a new framework, called SPAST, to generate high-quality stylized images with less inference time. Specifically, we design a novel Local-global Window Size Stylization Module (LGWSSM) to fuse style features into content features. Besides, we introduce a novel style prior loss, which can dig out the style priors from a pre-trained large-scale model into the SPAST and motivate the SPAST to generate high-quality stylized images with short inference time. We conduct abundant experiments to verify that our proposed method can generate high-quality stylized images and less inference time compared with the SOTA arbitrary style transfer methods.

Introduction

Arbitrary Style Transfer (AST) seeks to synthesize a hybrid image that merges the semantic content of one image with the artistic style of another. Historically, this field has been bifurcated by a significant trade-off between quality and efficiency. Small Model-based Methods (SMMs) offer real-time performance but often produce visually inconsistent results or "disharmonious" artifacts due to their limited capacity for understanding complex artistic textures. Conversely, Large-scale Model-based Methods (LMMs), such as those leveraging Stable Diffusion, produce high-quality aesthetics but are computationally expensive, often requiring seconds for a single inference. Furthermore, LMMs frequently struggle with structural fidelity, sometimes introducing "hallucinated" details that deviate from the original content.

The SPAST (Style Priors for Arbitrary Style Transfer) framework addresses these challenges by distilling the artistic knowledge of large-scale generative models into a lightweight architecture. By leveraging a novel Style Prior (SP) loss, SPAST achieves the aesthetic sophistication of diffusion models while maintaining the millisecond-level speeds and structural integrity of feed-forward networks.

Methodology

The SPAST framework is trained in two distinct stages, utilizing a specialized architectural module for feature fusion.

Stage One: Artistic Stable Diffusion (ASD)

The first stage involves fine-tuning a pre-trained Stable Diffusion model on the WikiArt dataset. To enable arbitrary style transfer, the model is modified to accept a style image as a reference. This is achieved through CLIP image embeddings and a learnable self-attention network, transforming the diffusion model into an "Artistic Stable Diffusion" (ASD) teacher that possesses deep priors for various artistic styles.

Stage Two: Distillation via Style Prior (SP) Loss

The core innovation of SPAST is the Style Prior (SP) loss. During the training of the smaller SPAST model, its output is passed to the frozen ASD teacher. The ASD calculates a "noise residual" at a specific timestep ($t=500$), which serves as a gradient signal. This signal guides the smaller model to adjust its pixel-level outputs to better align with the artistic priors of the large-scale model. This distillation process allows the SPAST model to "dig out" the knowledge of the diffusion model without requiring the diffusion model's presence during inference.

The two-stage SPAST training framework.

Local-global Window Size Stylization Module (LGWSSM)

To facilitate effective feature fusion, SPAST employs the LGWSSM. This module consists of two components:

  1. Local Window (LWSSM): This component performs region-wise matching, aligning specific content patches with the most stylistically relevant areas of the style image (e.g., matching a sky region to specific brushwork).
  2. Global Window (GWSSM): This component refines the overall image, ensuring global consistency in color and texture across the entire stylized output.

Experiments and Results

The SPAST model was trained using the MS-COCO dataset for content and WikiArt for style. Evaluation involved 600 generated samples across 20 content and 30 style images, compared against nine state-of-the-art (SOTA) methods.

Qualitative and Quantitative Performance

Qualitative results demonstrate that SPAST effectively captures complex artistic textures while strictly adhering to the content structure. Quantitatively, SPAST achieved the lowest Content Loss (4.521) and Style Loss (1.0928) among all tested methods.

Qualitative comparison of SPAST against state-of-the-art arbitrary style transfer methods.

Efficiency and Human Preference

SPAST demonstrates remarkable efficiency, with an inference time of 0.010 seconds for a 512x512 image. This represents a 3,200x speedup over diffusion-based methods like DiffuseIT (32.352s). In a user study, participants preferred SPAST's results 54.8% of the time, highlighting its superior balance of artistic quality and structural fidelity.

Discussion and Conclusion

A key finding in the development of SPAST was the sensitivity of the Style Prior loss to the diffusion timestep. The authors identified $t=500$ as the "Goldilocks zone": higher noise levels ($t \approx 1000$) tended to destroy content structure, while lower noise levels ($t \approx 100$) failed to provide sufficient stylistic guidance.

While SPAST offers significant improvements in speed and quality, it involves a substantial training overhead (approximately 80 hours total). Future work may focus on automating timestep selection or extending this style-prior distillation approach to other image-to-image translation tasks. In summary, SPAST provides a robust solution for high-quality, real-time arbitrary style transfer by effectively bridging the gap between large-scale generative priors and efficient feed-forward architectures.

Summary updated

Abstract

Given an arbitrary content and style image, arbitrary style transfer aims to render a new stylized image which preserves the content image's structure and possesses the style image's style. Existing arbitrary style transfer methods are based on either small models or pre-trained large-scale models. The small model-based methods fail to generate high-quality stylized images, bringing artifacts and disharmonious patterns. The pre-trained large-scale model-based methods can generate high-quality stylized images but struggle to preserve the content structure and cost long inference time. To this end, we propose a new framework, called SPAST, to generate high-quality stylized images with less inference time. Specifically, we design a novel Local-global Window Size Stylization Module (LGWSSM)tofuse style features into content features. Besides, we introduce a novel style prior loss, which can dig out the style priors from a pre-trained large-scale model into the SPAST and motivate the SPAST to generate high-quality stylized images with short inference time.We conduct abundant experiments to verify that our proposed method can generate high-quality stylized images and less inference time compared with the SOTA arbitrary style transfer methods.
BibTeX
@inproceedings{zhang2025spast,
  author = {Zhanjie Zhang and Quanwei Zhang and Junsheng Luan and Mengyuan Yang and Yun Wang and Lei Zhao},
  title = {{SPAST: Arbitrary Style Transfer with Style Priors via Pre-trained Large-scale Model}},
  booktitle = {Neural Networks},
  year = {2025},
  url = {https://openreview.net/forum?id=etvdyJPKSd},
}
Figure