Abstract
Text-driven style transfer in diffusion models often suffers from "semantic drift," where stylistic changes inadvertently alter the structural integrity of the source image. We present Adaptive Style Incorporation (ASI), a tuning-free method that decouples content and style at the feature level. By utilizing a dual-track Siamese Cross-Attention mechanism and adaptive masking based on feature covariance and Principal Component Analysis (PCA), ASI achieves superior stylistic transformation while preserving the original image's "skeleton." Experimental results demonstrate that ASI significantly outperforms existing state-of-the-art methods in both structural consistency and human preference.
Introduction
Text-to-image (T2I) diffusion models have enabled unprecedented creative flexibility. However, applying a specific style to an existing image via text prompts remains challenging. Standard methods often struggle with misalignment between text and image features, leading to unwanted changes in the subject's geometry, background layout, or fine structural details. For professional applications where composition is paramount, this lack of control is a critical limitation. ASI addresses this by moving style injection from the prompt level to the feature level, ensuring that stylistic attributes do not override structural foundations.
Methodology
ASI introduces two primary components to the diffusion pipeline, both of which are entirely tuning-free and compatible with pre-trained models like Stable Diffusion.
Siamese Cross-Attention (SiCA)
Traditional style transfer often concatenates content and style prompts, leading to feature interference. ASI replaces this with Siamese Cross-Attention (SiCA). This dual-track architecture uses a shared Query (representing the image's spatial structure) to simultaneously query two distinct sets of Keys and Values: one track for the content prompt (e.g., "a man") and another for the style prompt (e.g., "origami style"). This separation allows the model to process structural and stylistic information independently before blending.
Adaptive Content-Style Blending (AdaBlending)
To merge the dual tracks, ASI employs AdaBlending, which uses two intelligent masks to protect the image's core features:
- Attention-Head Mask ($M_{head}$): By calculating the statistical covariance between attention heads, the model identifies which heads are most effective for style transformation. Style injection is skipped for heads that show little difference between content and style, reducing unnecessary distortion.
- Spatial Mask ($M_{spatial}$): Using PCA on the attention features, ASI identifies the high-frequency edges or "skeleton" of the image. Style injection is restricted to non-structural regions, ensuring the "bones" of the original image remain untouched while the "skin" is modified.
Experimental Results
ASI was evaluated using real-world images (via DDIM inversion), generated images, and the ImageNet-R-TI2I benchmark. The method demonstrates remarkable versatility across a wide spectrum of artistic domains and photographic enhancements.
Quantitative and Qualitative Performance
- Structure Consistency: ASI achieved a Structure Distance of 0.016, a significant improvement over the next best method, Prompt-to-Prompt (P2P), which scored 0.053. This metric confirms ASI's ability to maintain the original layout.
- User Preference: In a study with 40 participants, ASI was preferred 70-78% of the time over other state-of-the-art methods like P2P, PnP, and FreeDiff.
- Photographic Enhancement: Beyond artistic styles, ASI can apply photographic effects like "Cinematic Lighting" or "Depth of Field" while keeping the subject perfectly preserved.
Discussion and Limitations
While ASI provides a robust solution for structure-consistent transfer, it faces certain challenges. The reliance on DDIM inversion for real-world images can occasionally result in the loss of extremely fine textures during the reconstruction process. Additionally, because the method is designed to protect structure, it is inherently less effective for styles that require intentional geometric distortion, such as Cubism. Finally, the real-time calculation of PCA and feature covariance adds a minor computational overhead compared to simpler, less precise methods.
Conclusion
Adaptive Style Incorporation (ASI) offers a powerful, tuning-free framework for text-driven style transfer. By decoupling content and style at the feature level and employing adaptive masking to protect structural integrity, ASI provides creators with the surgical control necessary for high-fidelity artistic transformation.
Summary updated