StyleAdapter: A Unified Stylized Image Generation Model

Zhouxia Wang, Xintao Wang, Liangbin Xie, Zhongang Qi, Ying Shan, Wenping Wang, Ping Luo

TL;DR

StyleAdapter is a unified, zero-shot stylized image generation model that eliminates the need for time-consuming per-style fine-tuning (such as LoRA or DreamBooth). By decoupling text and style processing and actively suppressing semantic leakage from reference images, it achieves an exceptional balance between text prompt control and artistic style fidelity.

What Problem Does This Paper Address?

While text-to-image (T2I) diffusion models like Stable Diffusion excel at generating images from textual prompts, text alone often lacks the descriptive power to capture intricate artistic styles, such as specific brushstrokes, textures, or paint mediums. To overcome this, recent methods utilize visual style references.

However, current state-of-the-art approaches like DreamBooth and LoRA require training or fine-tuning a separate model for every single new style. This per-style optimization is computationally expensive and impractical for real-time applications. Conversely, existing "zero-shot" or unified adapter methods suffer from two major flaws when given a style reference image:

  1. Loss of prompt controllability: The style features dominate the text features, causing the model to ignore the text prompt.
  2. Semantic leakage: The model inadvertently inherits the content (objects, layout) of the style reference image rather than just its artistic style.

StyleAdapter introduces a architecture designed to solve both problems simultaneously without requiring any per-style training.

Methods

StyleAdapter builds upon a frozen Stable Diffusion backbone and introduces two primary novel components:

  • Two-Path Cross-Attention (TPCA) Module: Instead of simply concatenating text and visual style features into a single condition vector, TPCA processes them through two parallel cross-attention paths within the UNet denoising blocks. The text prompt path is frozen to preserve original T2I controllability, while the style path is trained. The outputs are fused using a learnable, block-wise coefficient ($\lambda$), ensuring that the text prompt retains control over the layout and content while the style path injects artistic details.
  • Semantic Suppressing Vision Model (SSVM): To prevent the content of the style reference image from leaking into the generated output, the authors modify the CLIP vision encoder to strip away semantic layout information. SSVM achieves this through three strategies:
  • Removing the class embedding token ($E_{cls}$), which is heavily loaded with semantic classification data.
  • Patch-wise shuffling the reference image features by randomly shuffling the position embeddings ($E_{pos}$) before passing them to the vision encoder.
  • Utilizing multiple reference images of the same style but with completely different contents during training and inference.

The overall framework is illustrated below:

StyleAdapter Framework Overview

Additionally, a Style Embedding Module (StyEmb) uses a transformer block with a learnable embedding token to flexibly extract and map these suppressed vision features into the style path.

Datasets

  • Training: The model is trained on a 600K image-text pair subset of the LAION-AESTHETICS dataset. To simulate style references during training, the authors apply heavy data augmentations (cropping, resizing, flipping, rotation) to the target images to serve as style inputs.
  • Evaluation: The authors constructed a diverse evaluation benchmark consisting of 50 text prompts, 50 content images, and 8 distinct groups of style references collected from the internet (each containing 5 to 14 images), resulting in 400 unique evaluation pairs.

Key Insights and Ideas

  • Decoupled Attention is Mandatory: Naively mixing text and image tokens in a single cross-attention layer causes the network to prioritize visual tokens, drowning out the text prompt. Splitting them into distinct paths guarantees that the text prompt's authority over the image layout remains intact.
  • CLIP is Too Smart for Pure Style: Because CLIP vision models are optimized to understand what is in an image, their default features are inherently semantic. By intentionally breaking the spatial structure (via patch shuffling) and removing classification tokens, the authors force the network to only look at low-level artistic attributes like color distribution and texture.
  • Semantic Dilution via Multi-References: Providing multiple distinct images of the same style (e.g., a painting of a flower, a mountain, and a person in the same ink style) acts as a natural semantic filter. The network learns to ignore the varying contents and only extract the shared artistic medium.

Key Results

StyleAdapter demonstrates superior performance in balancing text alignment and stylization quality compared to both traditional style transfer methods and existing diffusion adapters.

Qualitative comparison with state-of-the-art methods
  • Quantitative Performance: In single-reference setups, StyleAdapter achieved a significantly lower Fréchet Inception Distance (FID) of 141.78 (indicating higher image quality) compared to ProSpect (160.66) and InST (177.91), while maintaining competitive text and style similarity scores.
  • User Preference: A subjective user study involving AIGC experts ranked StyleAdapter highest across all metrics: text similarity, style similarity, and overall image quality.
  • Downstream Compatibility: StyleAdapter can be seamlessly combined with spatial control networks like T2I-Adapter and ControlNet, allowing users to guide the generation process using sketches or structures while maintaining the target style.

Limitations and Future Work

Despite its strong generalization capabilities, StyleAdapter struggles with highly complex or rare artistic styles that are underrepresented in its general training data (such as complex transparency effects). In these specialized cases, intensive per-style fine-tuning methods like StyleDrop still hold an advantage.

The authors note that expanding the diversity of the training dataset and refining the algorithm to handle complex abstract styles are key directions for future work.

Summary updated

Abstract

This work focuses on generating high-quality images with specific style of reference images and content of provided textual descriptions. Current leading algorithms, i.e., DreamBooth and LoRA, require fine-tuning for each style, leading to time-consuming and computationally expensive processes. In this work, we propose StyleAdapter, a unified stylized image generation model capable of producing a variety of stylized images that match both the content of a given prompt and the style of reference images, without the need for per-style fine-tuning. It introduces a two-path cross-attention (TPCA) module to separately process style information and textual prompt, which cooperate with a semantic suppressing vision model (SSVM) to suppress the semantic content of style images. In this way, it can ensure that the prompt maintains control over the content of the generated images, while also mitigating the negative impact of semantic information in style references. This results in the content of the generated image adhering to the prompt, and its style aligning with the style references. Besides, our StyleAdapter can be integrated with existing controllable synthesis methods, such as T2I-adapter and ControlNet, to attain a more controllable and stable generation process. Extensive experiments demonstrate the superiority of our method over previous works.
BibTeX
@article{wang2025styleadapter,
  author = {Zhouxia Wang and Xintao Wang and Liangbin Xie and Zhongang Qi and Ying Shan and Wenping Wang and Ping Luo},
  title = {{StyleAdapter: A Unified Stylized Image Generation Model}},
  journal = {Int. J. Comput. Vis.},
  year = {2025},
  doi = {10.1007/s11263-024-02253-x},
}
Figure