StyleDrop: Text-to-Image Synthesis of Any Style

Kihyuk Sohn, Lu Jiang, Jarred Barber, Kimin Lee, Nataniel Ruiz, Dilip Krishnan, Huiwen Chang, Yuanzhen Li, Irfan Essa, Michael Rubinstein, Yuan Hao, Glenn Entis, Irina Blok, Daniel Castro ChinNeurIPS 2023 · OpenReview

Introduction

Pre-trained large text-to-image (T2I) models can synthesize impressive images given the right text prompts. However, they often struggle with semantic ambiguity and out-of-distribution effects when asked to replicate a specific arbitrary image style, texture, or material. If you ask for a "Van Gogh style," the model might yield a generic starry-night aesthetic, but it cannot easily replicate a unique, user-provided design pattern.

Existing personalization methods like DreamBooth or Textual Inversion were primarily designed to capture specific objects rather than styles. When applied to artistic styles, these methods frequently suffer from "content leakage"—accidentally copying the subject of the reference image (e.g., a house) instead of isolating the artistic style (e.g., the brushstrokes).

To address this, researchers introduce StyleDrop, a parameter-efficient method that enables T2I models to faithfully synthesize images in a specific style using as little as one reference image.

A grid demonstrating StyleDrop's ability to apply vastly different visual styles—from watercolor and 3D rendering to neon and wooden sculpture—to a consistent subject (a sloth using a laptop) based on single reference images (shown in insets).

Methodology: How StyleDrop Works

StyleDrop introduces a multi-pronged approach to style-tuning that departs from traditional diffusion-based methods:

1. The Transformer Backbone (Muse)

Unlike popular diffusion-based models (such as Stable Diffusion or Imagen), StyleDrop is built on Muse, a masked generative vision transformer. Muse operates on discrete visual tokens. The authors found that this discrete token approach is significantly more adept at learning fine-grained stylistic nuances from highly limited data than continuous diffusion processes.

2. Parameter-Efficient Fine-Tuning (PEFT)

To maintain computational efficiency, StyleDrop utilizes adapter tuning. Instead of updating the entire massive model, the researchers insert "adapters"—small sets of trainable parameters comprising less than 1% of the total model parameters—into the transformer layers.

3. Iterative Training with Feedback

Training a model on a single image notoriously leads to overfitting and content leakage. StyleDrop mitigates this through a novel iterative feedback loop:

  • Initial Training: An initial adapter is trained on the single reference image.
  • Generation: The model generates a large batch of synthetic images using that adapter.
  • Selection: The "best" images—those that successfully exhibit the desired style but feature entirely different content—are selected. This selection can be driven by automated CLIP scores or human feedback.
  • Distillation: A new adapter is retrained on this refined synthetic dataset. This process "distills" the style, teaching the model that the style is the constant while the subject matter is variable, effectively eliminating content leakage.

4. Compositional Generation

StyleDrop also supports "Style + Content" blending. A user can train one adapter for a specific style and another for a specific object (similar to DreamBooth). These adapters can then be combined at inference time to generate "your specific object in your specific style" without requiring joint training. Furthermore, using natural language descriptors (e.g., "in melting golden 3D rendering style") during training helps the model disentangle style from content better than relying on rare token identifiers alone.

Key Results and Insights

The research team evaluated StyleDrop using a diverse collection of style references (watercolor, 3D rendering, flat illustrations, etc.) and a subset of 190 prompts from the Parti benchmark to test compositional transfer.

  • Superior Fidelity and Preference: In user studies, StyleDrop on Muse was overwhelmingly preferred over existing methods. For style consistency, StyleDrop (using human feedback) scored a 60.9% preference rate, compared to a mere 9.7% for DreamBooth on Imagen.
  • Quantitative Edge: StyleDrop achieved a Style CLIP score of 0.705, significantly outperforming the 0.644 achieved by DreamBooth on Imagen. This indicates a much tighter alignment with the user-provided reference styles.
  • Efficiency: The method requires fine-tuning only a tiny fraction of parameters and can achieve high-quality, disentangled results with approximately 3 minutes of human feedback time.
  • Architecture Matters: A major takeaway is that the transformer-based Muse architecture significantly outperformed diffusion models in style fidelity, suggesting that discrete visual tokens may be inherently better suited for capturing specific textures and patterns.

Limitations and Future Directions

While StyleDrop represents a significant leap in stylistic control, the authors note several limitations. "Style" is inherently subjective; while CLIP scores provide a baseline, they are imperfect, and human feedback introduces bias and slows down the pipeline. Additionally, during iterative training, the style can sometimes "drift" or simplify if the feedback loop is not carefully managed.

Finally, the authors explicitly acknowledge the ethical implications of this technology. The ability to effortlessly copy an artist's unique style from a single image raises profound copyright and consent issues, underscoring the need for responsible deployment. Future work will explore expanding this few-shot transfer approach to even larger transformer models and incorporating more formal artistic attributes like composition and specific media types.

Summary updated

Abstract

Pre-trained large text-to-image models synthesize impressive images with an appropriate use of text prompts. However, ambiguities inherent in natural language, and out-of-distribution effects make it hard to synthesize arbitrary image styles, leveraging a specific design pattern, texture or material. In this paper, we introduce *StyleDrop*, a method that enables the synthesis of images that faithfully follow a specific style using a text-to-image model. StyleDrop is extremely versatile and captures nuances and details of a user-provided style, such as color schemes, shading, design patterns, and local and global effects. StyleDrop works by efficiently learning a new style by fine-tuning very few trainable parameters (less than 1\% of total model parameters), and improving the quality via iterative training with either human or automated feedback. Better yet, StyleDrop is able to deliver impressive results even when the user supplies only a *single* image specifying the desired style. An extensive study shows that, for the task of style tuning text-to-image models, StyleDrop on Muse convincingly outperforms other methods, including DreamBooth and textual inversion on Imagen or Stable Diffusion. More results are available at our project website: [https://styledrop.github.io](https://styledrop.github.io).
BibTeX
@inproceedings{sohn2023styledrop,
  author = {Kihyuk Sohn and Lu Jiang and Jarred Barber and Kimin Lee and Nataniel Ruiz and Dilip Krishnan and Huiwen Chang and Yuanzhen Li and Irfan Essa and Michael Rubinstein and Yuan Hao and Glenn Entis and Irina Blok and Daniel Castro Chin},
  title = {{StyleDrop: Text-to-Image Synthesis of Any Style}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=KoaFh16uOc},
}
Figure