StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Synthesis

Peter Schaldenbrand, Zhixuan Liu, Jean OharXiv 2021 · arXiv

Abstract

StyleCLIPDraw is an evolution of the CLIPDraw architecture that enables simultaneous control over content and artistic style in text-to-drawing synthesis. By coupling a CLIP-based content loss with a STROTSS-based style loss, the model ensures that stylistic elements influence the fundamental geometry and stroke composition of the generated vector drawings. This approach overcomes the limitations of decoupled style transfer, where style is typically applied as an additive post-processing filter.

Introduction

Text-to-image synthesis has seen significant progress with models like DALL-E and CLIPDraw. However, providing precise artistic direction remains a challenge. While a user can specify content (e.g., "a monkey playing guitar"), controlling the specific artistic style—such as charcoal sketching or minimalist line art—often requires post-processing. StyleCLIPDraw addresses this by "coupling" style and content during the generation process, allowing style to guide the synthesis from the initial brushstrokes.

Methods

StyleCLIPDraw utilizes the CLIPDraw framework, which represents images as a collection of differentiable Bézier curves. This vector-based approach provides a natural inductive bias for sketch-like generation. The optimization process is driven by a dual-loss function:

  1. Content Loss (CLIP): Employs the pre-trained CLIP model to minimize the cosine distance between the text prompt embedding and the augmented image embedding.
  2. Style Loss (STROTSS): Uses the STROTSS algorithm to compare feature maps from a VGG-16 network for both the generated drawing and a reference style image.

Because the renderer is differentiable, the model can backpropagate gradients from both losses to update the parameters of the Bézier curves (coordinates, color, and opacity) in real-time. This ensures that style influences the very shapes and structures of the composition, rather than just surface textures.

Results and Key Insights

The coupled approach demonstrates a superior ability to integrate style into the drawing's structure compared to decoupled baselines. A key insight is that style and content are inseparable in art; when style loss is applied during the drawing process, the model changes the geometry of the strokes.

Figure 6: A man walking a dog, synthesized in a minimalist line-art style. The model adapts the stroke geometry to match the stylistic reference.

In the "man walking a dog" example, StyleCLIPDraw produces a genuine line drawing composed of minimalist outlines. In contrast, decoupled methods often fail to alter the underlying shaded structure of the base content. The model is robust across various prompts, maintaining content recognizability even under heavy stylistic constraints.

Figure 3: A monkey playing guitar, showing the model's capacity to represent complex scenes while adhering to stylistic constraints.

Discussion and Limitations

While StyleCLIPDraw effectively couples style and content, it occasionally produces abstract or messy compositions that lack realism. Furthermore, as it relies on CLIP, it inherits the biases present in CLIP's large-scale training data. Future work involves scaling to more complex scenes and improving optimization stability to ensure more consistent results across diverse prompts and styles.

Summary updated

Abstract

Generating images that fit a given text description using machine learning has improved greatly with the release of technologies such as the CLIP image-text encoder model; however, current methods lack artistic control of the style of image to be generated. We introduce StyleCLIPDraw which adds a style loss to the CLIPDraw text-to-drawing synthesis model to allow artistic control of the synthesized drawings in addition to control of the content via text. Whereas performing decoupled style transfer on a generated image only affects the texture, our proposed coupled approach is able to capture a style in both texture and shape, suggesting that the style of the drawing is coupled with the drawing process itself. More results and our code are available at https://github.com/pschaldenbrand/StyleCLIPDraw
BibTeX
@misc{schaldenbrand2021styleclipdraw,
  author = {Peter Schaldenbrand and Zhixuan Liu and Jean Oh},
  title = {{StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Synthesis}},
  year = {2021},
  eprint = {2111.03133},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2111.03133},
}
Figure