Abstract: We present Diffusion in Style, a simple method to adapt Stable Diffusion to any desired style, using only a small set of target images. It is based on the key observation that the style of the images generated by Stable Diffusion is tied to the initial latent tensor. Not adapting this initial latent tensor to the style makes fine-tuning slow, expensive, and impractical, especially when only a few target style images are available. In contrast, fine-tuning is much easier if this initial latent tensor is also adapted. Our Diffusion in Style is orders of magnitude more sample-efficient and faster. It also generates more pleasing images than existing approaches, as shown qualitatively and with quantitative comparisons.
Introduction
Large-scale text-to-image models like Stable Diffusion have revolutionized content creation, yet they often struggle to adhere to specific, niche artistic styles. Users frequently resort to "prompt engineering"—appending phrases like "in the style of..."—which often fails to capture the subtle nuances of a particular aesthetic.
Traditional alternatives, such as full-model fine-tuning, typically require thousands of images and significant computational resources. Moreover, aggressive fine-tuning often leads to "catastrophic forgetting," where the model loses its ability to generate diverse objects or follow complex prompts. The authors of "Diffusion in Style" identify a core bottleneck: Stable Diffusion begins its generation process from a standard Gaussian noise distribution. This "neutral" starting point is biased toward the natural image domain, making it computationally expensive to "push" the model toward highly stylized outputs during denoising.
Methodology: Adapting the Latent Prior
The researchers propose a two-step adaptation process that leverages the stylistic information encoded in the initial latent tensor. By shifting the starting point of the diffusion process closer to the target style's "latent territory," the model requires far less modification to produce consistent results.
- Adapting the Noise Distribution: Instead of starting with standard Gaussian noise ($N(0, I)$), the method calculates a style-specific distribution. A small set of target style images is passed through the Variational Auto-Encoder (VAE) encoder to compute the element-wise mean ($\mu_{style}$) and standard deviation ($\sigma_{style}$) of these latent representations.
- Efficient Fine-tuning: The U-Net is fine-tuned for a very short duration (typically 1,000 iterations). Crucially, during this training, noise is sampled from the new style-specific distribution rather than the standard one.
For extremely low-data regimes (e.g., only 3 images), the distribution is further simplified to be spatially constant to prevent overfitting to specific compositions.
Experimental Results
The method was evaluated across diverse styles, including Anime Sketches, 1940s comic panels ("48 Famous Americans"), and Vincent van Gogh’s "Starry Night."
Sample Efficiency and Speed
Diffusion in Style is remarkably efficient. While traditional methods might require over 15,000 iterations, this approach succeeds in just 1,000. Fine-tuning typically takes less than 20 minutes on a single NVIDIA Tesla V100 GPU, with extreme low-data adaptations (like the 3-image "Starry Night" set) completing in under 5 minutes.
Qualitative and Quantitative Performance
Quantitative evaluation using Pareto curves (comparing CLIP scores for prompt alignment vs. Normalized FID for style matching) showed that Diffusion in Style consistently outperforms existing methods. Qualitatively, the model excels at applying new styles to objects never seen in the training set—such as a "panda making latte art" in a comic style—demonstrating that it retains its general knowledge while adopting the target aesthetic.
Discussion and Limitations
Despite its efficiency, the method relies on Classifier-Free Guidance, and the optimal guidance weight can vary by style and prompt, sometimes requiring manual tuning. Additionally, like the base Stable Diffusion model, it struggles with legible text rendering in stylized contexts, such as comic book speech bubbles.
Future work may explore finding initial latents that are style-specific yet close to the original distribution, potentially enabling style adaptation without any fine-tuning of the U-Net weights. Currently, Diffusion in Style represents a significant step forward in making high-quality, personalized generative AI accessible with minimal data and compute.
Summary updated