TL;DR
This paper introduces FastCLIPstyler, a novel text-based image style transfer model that achieves impressive stylization results in a single forward pass, eliminating the computationally expensive runtime optimization required by previous methods like CLIPstyler. They also present EdgeCLIPstyler, a lightweight variant optimized for resource-constrained devices, making advanced style transfer accessible on edge platforms.
What Problem Does This Paper Address?
Traditional image style transfer methods require a reference style image to guide the stylization process. While recent advancements have enabled text-based image style transfer (LDAST), allowing users to describe the desired style using natural language, existing methods like CLIPstyler suffer from a significant drawback: they require a lengthy optimization process at inference time for each new style query. This makes them impractical for real-time applications and user-friendly interfaces. Furthermore, some LDAST models struggle with generalization to unseen prompts or are too computationally demanding for deployment on edge devices. This paper aims to bridge this gap by developing efficient, generalizable, and deployable LDAST models.
Methods
The core innovation of FastCLIPstyler lies in its departure from runtime optimization. Instead of training a model from scratch for each query, FastCLIPstyler leverages a pre-trained, purely vision-based style transfer network (specifically, the style application network from Ghiasi et al. [10]). The key is a newly introduced "text-style prediction network," a compact feed-forward network that takes CLIP text embeddings as input and predicts a 100-dimensional style representation compatible with the pre-trained style application network. This network is trained on a self-generated dataset of text prompts and their corresponding style embeddings, effectively learning a mapping from language to style representations.
To address the issue of style embeddings potentially falling outside the valid input region of the pre-trained network, a "distribution loss" ($L_{dis}$) is incorporated. This loss penalizes predictions that deviate significantly from the distribution of style embeddings derived from real-world datasets (PBN and DTD). The overall loss function for training the text-style prediction network combines this distribution loss with CLIP-based losses ($L_{dir}$ and $L_{patch}$) for style transfer quality.
For edge deployment, EdgeCLIPstyler is introduced. This variant uses a more resource-efficient text embedder, Sentence-BERT paraphrase-albert-v2, during inference, while still leveraging CLIP for training supervision. This architectural tweak allows for LDAST on low-power devices.
Datasets
The paper highlights a significant advantage: their approach does not require a pre-existing labeled image dataset for LDAST training. Instead, they generate their own dataset. This involves:
- Initial Prompt Generation: Combining keywords for colors, textures, art styles, and objects to create a dataset of 4,300 prompts.
- ChatGPT Enhancement: Utilizing ChatGPT to generate an additional 1,500 style prompts, increasing dataset diversity and covering aspects like nature, emotional states, and examples similar to the ArtEmis dataset.
For quantitative evaluation, they create a dataset of 80 prompts, including general prompts and those inspired by the DTD and ArtEmis datasets. They also use semi-ground truth (Semi-GT) results generated by AdaAttn [17] from style images for metrics like SSIM, Percept, and FAD.
Key Insights and Ideas
- Decoupling Style Representation from Runtime Optimization: The most significant insight is that by learning a mapping from text embeddings to a fixed-dimension style representation (used by a pre-trained vision model), runtime optimization can be entirely eliminated. This dramatically speeds up inference.
- Self-Generated Datasets for LDAST: The ability to create a training dataset for LDAST solely from text prompts, without needing paired image-text data or reference style images, is a major practical advantage, reducing data collection costs and effort.
- Edge-Compatible LDAST: The development of EdgeCLIPstyler demonstrates that advanced LDAST can be made feasible on resource-constrained devices, opening up new application possibilities for mobile and embedded systems.
- Distribution Loss for Robustness: The introduction of the distribution loss is crucial for ensuring that the predicted style embeddings are within the expected range of the pre-trained vision model, leading to more stable and visually pleasing results.
Key Results
- Speed: FastCLIPstyler is up to 730x faster than CLIPstyler on a powerful GPU. On edge devices (CPU-only), it runs three times faster than CLVA. EdgeCLIPstyler is approximately 15x faster than CLVA on resource-constrained devices.
- Quality: The models achieve superior stylization quality, maintaining image integrity and avoiding artifacts seen in CLIPstyler outputs (Figure 5). They demonstrate good generalization to unseen prompts, outperforming CLVA in this regard.
- Edge Performance: EdgeCLIPstyler can run on a Raspberry Pi 3B+ in 15 seconds, a feat not possible with CLIPstyler, FastCLIPstyler, or CLVA due to memory constraints.
- Quantitative Metrics: FastCLIPstyler achieves the highest SSIM and a comparable FAD score to the Semi-GT. While CLIPstyler matches text prompts more closely (VLS), FastCLIPstyler shows strong stylization performance with lower Percept loss. EdgeCLIPstyler achieves the lowest overall similarity distance (FAD) with the Semi-GT. Human evaluation also favored their models.
Limitations and Future Work
- Reliance on Pre-trained Vision Models: The current approach relies on adopting a pre-existing vision-based style transfer model with an explicit low-dimensional style representation. The authors acknowledge that recent research is moving towards models that learn high-dimensional implicit style representations using attention networks, which their current framework might not directly accommodate.
- Monochromatic Tendency: The models can sometimes produce more monochromatic images when a specific color is implied in the prompt (e.g., "brown fur").
- EdgeCLIPstyler vs. FastCLIPstyler: EdgeCLIPstyler occasionally falls behind FastCLIPstyler in quality for certain prompts.
- Future Work: The paper suggests exploring models that learn high-dimensional implicit style representations, potentially by incorporating attention mechanisms, to further enhance the model's capabilities and adaptability.
Summary updated