TL;DR
This paper introduces QwenStyle, the first content-preserving style transfer model built on the Qwen-Image-Edit framework. It tackles the challenge of maintaining content integrity while applying diverse styles to images, achieving state-of-the-art results through a novel curriculum continual learning approach and a carefully curated dataset.
What Problem Does This Paper Address?
Style transfer, the process of applying the artistic style of one image to the content of another, has seen significant advancements. However, a persistent challenge, especially for diffusion transformer (DiT) models, is content-preserving style transfer. This means not only applying a new style but doing so while ensuring the original content's key features, such as object identities, poses, and overall structure, remain intact. Existing DiT models often struggle with this due to the inherent entanglement of content and style features within their internal representations. This paper aims to bridge this gap by developing a DiT-based model that excels at both style transfer and content preservation.
Methods
The core of QwenStyle lies in its training methodology and dataset construction, built upon the Qwen-Image-Edit model.
- Qwen-Image-Edit Foundation: The model leverages Qwen-Image-Edit, a powerful text-to-image and image editing model, which already possesses strong content preservation and style customization capabilities.
- Curriculum Continual Learning: To address the complexities of content preservation and generalization, the authors propose a three-stage curriculum learning approach:
- Stage 1 (D1): Initial training on a clean, curated dataset (Dcollected) to activate basic content-preserving style transfer. This stage reveals limitations in preserving subtle content details and generalizing to unseen styles.
- Stage 2 (D2): Fine-tuning with a dataset enriched with high-content-similarity triplets. This stage significantly improves the preservation of subtle content characteristics.
- Stage 3 (D3): Further fine-tuning using a mixture of high-content-similarity data and a low ratio of synthetic, diverse style triplets (Dsynthetic). This stage balances content preservation with improved generalization to out-of-distribution styles.
- Synthetic Triplet Generation: To overcome the scarcity of diverse style triplets, the authors developed a framework to synthesize triplets from existing images in the wild. This involves using image editing models and style extraction techniques to create content and style references.
Datasets
- Dcollected: A dataset of high-quality, limited-style triplets collected from various sources, including GPT-4O and open-source LoRAs. This dataset contains approximately 300,000 triplets across 30 style categories.
- Dsynthetic: A larger dataset of approximately 1 million synthetic triplets generated using a novel framework. This dataset is crucial for improving generalization to unseen styles. The synthesis process involves creating content and style references from stylized images.
Key Insights and Ideas
- The Challenge of DiTs: Unlike some previous models where content and style could be more easily disentangled, DiTs present a unique challenge for content-preserving style transfer due to their internal feature entanglement.
- Curriculum Learning is Key: A staged learning approach is essential. Starting with clean data, then focusing on content similarity, and finally introducing diverse synthetic data, allows the model to progressively learn and refine its capabilities without catastrophic forgetting.
- Synthetic Data for Generalization: While clean, curated data is good for initial learning, synthetic data, even if "noisy" in terms of perfect content preservation, is vital for achieving broad generalization to unseen styles.
- Prompt Engineering Matters: While QwenStyle V1 supports custom prompts, a default prompt emphasizing style transfer while preserving content is recommended for optimal results.
Key Results
- State-of-the-Art Performance: QwenStyle V1 achieves state-of-the-art performance across three core metrics: style similarity, content consistency, and aesthetic quality.
- Quantitative Improvements: In Table 1, QwenStyle V1 (Style-CCL) shows a superior CSD score (0.577) for style similarity compared to OmniStyle (0.447), OmniGen-v2 (0.462), and DreamO (0.402). It also achieves the highest Aesthetic Score (6.317).
- Qualitative Superiority: Figure 3 visually demonstrates QwenStyle V1's ability to transfer styles effectively while preserving complex content details, outperforming other state-of-the-art models.
Limitations and Future Work
- Experimental Stage: The authors acknowledge that QwenStyle V1 is an experimental attempt and that there is considerable potential for improvement.
- Generalization on Out-of-Distribution Styles: While improved, the generalization capability on truly out-of-distribution styles is still an area for further enhancement.
- Content Reference Realism: The paper notes that the realism converter used for generating content references in the synthetic triplet pipeline was based on FLUX-dev due to time constraints. Future work will involve training a more robust realism converter with Qwen-Image.
Summary updated