Abstract
StyleCLIPDraw is an evolution of the CLIPDraw architecture that enables simultaneous control over content and artistic style in text-to-drawing synthesis. By coupling a CLIP-based content loss with a STROTSS-based style loss, the model ensures that stylistic elements influence the fundamental geometry and stroke composition of the generated vector drawings. This approach overcomes the limitations of decoupled style transfer, where style is typically applied as an additive post-processing filter.
Introduction
Text-to-image synthesis has seen significant progress with models like DALL-E and CLIPDraw. However, providing precise artistic direction remains a challenge. While a user can specify content (e.g., "a monkey playing guitar"), controlling the specific artistic style—such as charcoal sketching or minimalist line art—often requires post-processing. StyleCLIPDraw addresses this by "coupling" style and content during the generation process, allowing style to guide the synthesis from the initial brushstrokes.
Methods
StyleCLIPDraw utilizes the CLIPDraw framework, which represents images as a collection of differentiable Bézier curves. This vector-based approach provides a natural inductive bias for sketch-like generation. The optimization process is driven by a dual-loss function:
- Content Loss (CLIP): Employs the pre-trained CLIP model to minimize the cosine distance between the text prompt embedding and the augmented image embedding.
- Style Loss (STROTSS): Uses the STROTSS algorithm to compare feature maps from a VGG-16 network for both the generated drawing and a reference style image.
Because the renderer is differentiable, the model can backpropagate gradients from both losses to update the parameters of the Bézier curves (coordinates, color, and opacity) in real-time. This ensures that style influences the very shapes and structures of the composition, rather than just surface textures.
Results and Key Insights
The coupled approach demonstrates a superior ability to integrate style into the drawing's structure compared to decoupled baselines. A key insight is that style and content are inseparable in art; when style loss is applied during the drawing process, the model changes the geometry of the strokes.
In the "man walking a dog" example, StyleCLIPDraw produces a genuine line drawing composed of minimalist outlines. In contrast, decoupled methods often fail to alter the underlying shaded structure of the base content. The model is robust across various prompts, maintaining content recognizability even under heavy stylistic constraints.
Discussion and Limitations
While StyleCLIPDraw effectively couples style and content, it occasionally produces abstract or messy compositions that lack realism. Furthermore, as it relies on CLIP, it inherits the biases present in CLIP's large-scale training data. Future work involves scaling to more complex scenes and improving optimization stability to ensure more consistent results across diverse prompts and styles.
Summary updated