Arbitrary Style Guidance for Enhanced Diffusion-Based Text-to-Image Generation

Zhihong Pan, Xin Zhou, Hao TianWACV 2023

TL;DR

This paper introduces a novel method for controlling the artistic style of images generated by diffusion models, allowing users to specify arbitrary styles using a reference image or even generate diverse styles without one. This approach enhances existing text-to-image models by enabling fine-grained style control during inference, without requiring model retraining.

What Problem Does This Paper Address?

Current diffusion-based text-to-image models, while impressive, struggle with precise artistic style control. While they can generate images in broad categories like "oil painting" or "pencil drawing," specifying finer details like brush strokes, color palettes, or the unique style of a particular artist is difficult. This is because style is often sampled from a conditional distribution, leading to a bias towards certain styles present in the training data. This paper aims to bridge this gap by enabling users to guide the generation process with arbitrary artistic styles, either from a reference image or through self-guided exploration, without needing to retrain the underlying diffusion model.

Methods

The core of the proposed method lies in a novel style guidance technique applied during the reverse diffusion process. Instead of relying on traditional classifier guidance or CLIP guidance, which focus on class labels or text-image similarity, this approach uses style features extracted from images.

The authors propose two main guidance strategies:

  • Supervised Style Guidance: This method uses a reference image to guide the generation towards a specific artistic style. The key innovation here is applying the guidance to the "noise-free" estimate of the image ($x_0$) rather than the noisy intermediate step ($x_t$). This is achieved by calculating the gradient of the style difference between the generated image's style features and the reference image's style features. They also introduce an adaptive guidance scale that adjusts based on the noise level, improving effectiveness.
  • Self Style Guidance: This strategy allows for generating diverse styles without a reference image.
  • Contrastive Self Guidance: This aims to increase style variations within a batch of images generated from the same text prompt. It encourages style differences among the generated samples.
  • Synonymous Self Guidance: This method is designed to generate a series of images with a shared, consistent style, useful for applications like creating stamps or posters. It uses a "mixed style feature" derived from multiple images to guide the generation.

Style features are extracted using instance normalization statistics from VGG-19 layers, similar to the AdaIN method, but with optimized weights for guidance.

Datasets

The paper does not explicitly mention training datasets. For evaluation, they use a set of 12 random artworks from WikiArt as style references and organize text inputs into 5 content categories (dogs, flowers, wonders of the world, American landmarks, and general places). They also mention using OpenAI's filtered GLIDE model as their baseline.

Key Insights and Ideas

  • Style Guidance at Inference Time: The ability to guide style during inference without retraining is a significant advantage, making the method broadly applicable to existing diffusion models.
  • Guidance on $x_0$ vs. $x_t$: Applying guidance to the estimated clean image ($x_0$) rather than the noisy intermediate step ($x_t$) is crucial for avoiding confusion between noise and style characteristics, especially when dealing with complex artistic styles.
  • Adaptive Guidance Scale: The insight that guidance scale should adapt to the noise level (higher scale at lower noise) significantly improves the trade-off between style fidelity and text-image similarity.
  • Self-Guidance for Diversity and Consistency: The development of both contrastive and synonymous self-guidance offers flexible ways to control style diversity and consistency, addressing different user needs.
  • Instance Normalization for Style Features: Using instance normalization statistics proves effective for capturing style information, offering a cleaner alternative to Gram matrices in this context.

Key Results

  • Supervised Style Guidance: The method successfully generates images with desired artistic styles from reference images, achieving better consistency with the style reference than two-step methods (applying neural style transfer after unguided generation).
  • Self Style Guidance:
    • Contrastive Self Guidance increases style variations compared to unguided sampling.
  • Synonymous Self Guidance generates images with a shared style and maintains high text-image similarity.
  • Quantitative Evaluation: While FID is not applicable due to the lack of ground truth, CLIP scores are used to assess text-image similarity. The paper shows that their method achieves good CLIP scores while also maintaining style fidelity. Ablation studies (Table 1, Fig. 4) demonstrate that their proposed optimal settings (guidance on $x_0$, adaptive scale, MAE for distance, optimal weights) lead to the best overall quality.

Limitations and Future Work

  • Style Feature Robustness: The paper acknowledges that the robustness of style feature functions to noisy images is not fully understood, which motivates their proposed guidance on $x_0$.
  • Computational Cost: While the method is applied at inference time, diffusion models are inherently computationally intensive.
  • Future Work: The authors suggest exploring more advanced style features, such as context-aware ones, for even richer style control. They also propose applying their method to other diffusion model architectures and exploring its use in generating more complex content.

Summary updated

BibTeX
@inproceedings{pan2023arbitrary,
  author = {Zhihong Pan and Xin Zhou and Hao Tian},
  title = {{Arbitrary Style Guidance for Enhanced Diffusion-Based Text-to-Image Generation}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2023},
}
Figure