StyleAdapter: A Unified Stylized Image Generation Model without Test-Time Fine-Tuning

Zhouxia Wang, Xintao Wang, Liangbin Xie, Zhongang Qi, Ying Shan, Wenping Wang, Ping LuoSubmission to ICLR 2024 · Withdrawn

TL;DR

StyleAdapter introduces an inference-time, fine-tuning-free framework for stylized text-to-image generation. By isolating style and prompt processing into a Two-Path Cross-Attention mechanism and stripping semantic content from reference images via a Semantic Suppressing Vision Model, it produces high-fidelity stylized images in a single pass without test-time retraining.

What Problem Does This Paper Address?

Current approaches to personalized, style-guided text-to-image (T2I) synthesis—such as DreamBooth, LoRA, and StyleDrop—yield strong results, but require fine-tuning or inverting the model for every new style. This test-time optimization is computationally expensive and slow for interactive applications.

Conversely, zero-shot conditioning (e.g., extracting image embeddings with CLIP and concatenating them with text prompts) tends to fail in two ways:

  1. Prompt Disregard: Visual tokens dominate cross-attention layers, causing the model to ignore textual prompt instructions.
  2. Semantic Leakage: Image encoders bind artistic style (textures, brushwork, palette) closely with semantic content (identities, objects). As a result, the generated image borrows both the style and the subject matter of the reference image rather than honoring the text prompt.

StyleAdapter aims to create a single, unified model capable of adapting to arbitrary visual styles on the fly while retaining strict prompt control.

Reference Style Images ──► [ SSVM (Shuffle + Drop E_cls) ] ──► [ StyEmb ] ──► Style Cross-Attention ─┐
                                                                                                        ├── (Scaled by λ) ──► Latent Diffusion
Text Prompt            ──► [ CLIP Text Encoder           ] ─────────────────► Text Cross-Attention  ─┘

Methods

Built on top of Stable Diffusion (SD v1.5), StyleAdapter freezes the base generator and CLIP encoders, adding two modular components:

  • Two-Path Cross-Attention (TPCA): Instead of feeding concatenated prompt and style embeddings into a single cross-attention layer, TPCA separates them. The original frozen SD cross-attention continues processing text features ($f_t$), while a parallel, trainable style cross-attention processes visual style features ($f_s$). Their outputs merge back into spatial features via a learnable scaling factor $\lambda$: $$\hat{y} = \text{Attention}(Q_t, K_t, V_t) + \lambda \cdot \text{Attention}(Q_s, K_s, V_s)$$
  • Style Embedding Module (StyEmb): A 3-block transformer where a learnable query token ($f_m$) extracts aggregated style features from visual representations, accommodating a flexible number of reference images.
  • Semantic Suppressing Vision Model (SSVM): To stop reference objects from leaking into the generated output, SSVM filters out high-level semantics through three steps:
  • Discarding the Class Token: Omitting CLIP ViT's $E_{cls}$ token, which primarily encodes global semantic class identities.
  • Patch-wise Shuffling: Randomly permuting the spatial position embeddings ($E_{pos}$) before feeding patches into the vision encoder, breaking spatial layout while preserving local texture and color statistics.
  • Multi-Reference Invariance: Conditioning on multiple images that share a style but depict different objects (e.g., a person, a landscape, a building in the same watercolor medium), encouraging the network to discard inconsistent semantic subjects.
  • Compatibility with Structural Control: StyleAdapter can natively combine with control models like T2I-Adapter and ControlNet to guide generation with sketches or edges.

Datasets

  • Training: A 600K image-text subset of LAION-Aesthetics. Multi-reference style inputs during training were constructed synthetically from training images using diverse data augmentations (crops, rotations, horizontal flips).
  • Evaluation Benchmark: A newly constructed test set consisting of 50 diverse prompts (generated via ChatGPT and curated), 50 content images generated via SD (seed 993), and 8 reference style sets (each containing 5–14 web-sourced images), totaling 400 test combinations.

Key Insights and Ideas

  • Attention Starvation Explains Lost Prompt Control: Attention map visualizations revealed that concatenating text and image tokens causes style features to absorb attention weights in the diffusion U-Net, starving prompt keywords. Separating attention into two distinct computational paths immediately restores text adherence.
  • Spatial Shuffling Eliminates Object Contamination: Semantic meaning is heavily tied to spatial arrangement. Shuffling patch positions disrupts high-level object concepts (e.g., turning a human face into incoherent parts) while leaving low-level style cues (e.g., ink washes, pencil hatching) intact for the attention layers.
  • Semantic Diversity Acts as a Regularizer: Providing multiple style references with varied content forces the attention mechanism to extract the common visual denominator—the style—and discard the variable object semantics.

Key Results

  • Balancing Text and Style: In multi-reference evaluations, StyleAdapter achieved higher text alignment than fine-tuned LoRA (Text-Sim: 0.2448 vs. 0.2390) and Textual Inversion (0.1492), while maintaining competitive style transfer (Style-Sim: 0.9031 vs. LoRA's 0.9034) without any test-time training.
  • Image Quality: StyleAdapter produced competitive visual quality with an FID of 140.97 (multi-reference), outperforming base SD (189.34) and InST (153.45).
  • User Preference: In a subjective evaluation by 24 professional AIGC users over 2,520 votes, StyleAdapter was preferred across all criteria:
  • Text Similarity: 42.74% preference (vs. LoRA's 28.69%)
  • Style Similarity: 39.76% preference (vs. LoRA's 18.81%)
  • Image Quality: 42.38% preference (vs. LoRA's 32.38%)

Limitations and Future Work

  • Gap with Per-Style Specialization: Because StyleAdapter is a generalized model operating without test-time optimization, specialized models trained exclusively on a single style (like heavily fine-tuned LoRA checkpoints) can still occasionally match nuanced style characteristics better.
  • Pre-trained Model Biases: The system inherits the standard generative artifacts and distribution biases of its base model, Stable Diffusion v1.5.

Summary updated

Abstract

This work focuses on generating high-quality images with specific style of reference images and content of provided textual descriptions. Current leading algorithms, i.e., DreamBooth and LoRA, require fine-tuning for each style, leading to time-consuming and computationally expensive processes. In this work, we propose StyleAdapter, a unified stylized image generation model capable of producing a variety of stylized images that match both the content of a given prompt and the style of reference images, without the need for test-time fine-tuning. It introduces a two-path cross-attention (TPCA) module to separately process style information and textual prompt, which cooperate with a semantic suppressing vision model (SSVM) to suppress the semantic content of style images. In this way, it can ensure the controllability of the prompt over the content of the generated images while mitigating the negative impact of semantic information in style references. Besides, our StyleAdapter can be integrated with existing controllable synthesis methods, such as T2I-adapter and ControlNet, to attain a more controllable and stable generation process. Extensive experiments demonstrate the superiority of our method over previous works.
BibTeX
@article{wang2025styleadapter,
  author = {Zhouxia Wang and Xintao Wang and Liangbin Xie and Zhongang Qi and Ying Shan and Wenping Wang and Ping Luo},
  title = {{StyleAdapter: A Unified Stylized Image Generation Model without Test-Time Fine-Tuning}},
  journal = {Int. J. Comput. Vis.},
  year = {2025},
  doi = {10.1007/s11263-024-02253-x},
}
Figure