TL;DR
This paper introduces "Attention Distillation" (AD), a novel loss function that leverages self-attention features from pre-trained diffusion models to transfer visual characteristics (style, appearance, texture) from a reference image to synthesized images. AD can be used for direct optimization or integrated into the diffusion sampling process, enabling faster and more versatile image generation across various tasks.
What Problem Does This Paper Address?
Synthesizing new images that capture the style, appearance, or texture of a reference image has been a long-standing challenge. Traditional methods often struggle with complex textures or abstract styles. While recent diffusion models show promise by using self-attention features to represent visual characteristics, existing "plug-and-play" approaches using these features face limitations: a "domain gap" between reference and target images, error accumulation during iterative sampling, and architectural constraints that limit the influence of these features. This paper aims to overcome these limitations by proposing a more effective way to distill and transfer visual characteristics.
Methods
The core contribution is the Attention Distillation (AD) loss. Instead of directly using self-attention features as attributes, AD calculates a loss based on the difference between an "ideal" attention output (generated by combining target queries with reference keys/values) and the "current" attention output (from the target branch). This loss is computed using the L1 distance and is backpropagated in the latent space to optimize the synthesized image.
Key techniques include:
- Attention Distillation (AD) Loss: Formulated as $L_{AD} = ||Self\text{-}Attn(Q, K, V) – Self\text{-}Attn(Q, K_s, V_s) ||_1$. This loss encourages the model to learn a better alignment between target queries and reference keys/values.
- Content-Preserving Optimization: AD loss is combined with a content loss ($L_{content} = ||Q - Q_c||_1$) to preserve the structural elements of a content image while transferring style or appearance.
- Attention Distillation Guided Sampling: AD loss is integrated into the diffusion sampling process as an improved Classifier Guidance. This allows for faster synthesis and enables style-specific text-to-image generation.
- Improved VAE Decoding: Optionally fine-tuning the VAE decoder on the reference image can enhance reconstruction quality for high-frequency details.
Datasets
The paper does not explicitly mention training new datasets. Instead, it leverages pre-trained diffusion models like Stable Diffusion v1.5 and SDXL. The experiments utilize various images for style transfer, appearance transfer, text-to-image generation, and texture synthesis tasks, drawing from existing benchmarks and common image datasets implicitly used by these pre-trained models.
Key Insights and Ideas
- Distillation over Direct Use: The key insight is that directly using self-attention features as plug-and-play attributes is suboptimal. Distilling the difference between ideal and current attention outputs provides a more effective signal for transferring visual characteristics.
- Backpropagation is Crucial: Optimizing the latent space via backpropagation through the AD loss allows information to flow more comprehensively, leading to better stylization compared to methods that only modify features at specific layers.
- Unified Framework: The AD loss and its guided sampling variant provide a unified approach applicable to a wide range of tasks, from artistic style transfer to controlled texture synthesis and text-to-image generation.
- Guided Sampling for Efficiency: Integrating AD loss into the sampling process significantly accelerates synthesis while maintaining high quality, making it practical for real-time applications.
Key Results
- Superior Visual Quality: Experiments demonstrate that AD loss achieves notably superior results in transferring style, appearance, and texture compared to existing methods like KV-injection, StyleID, StyTR², Cross-Image Attention, and others.
- Faster Synthesis: The Attention Distillation Guided Sampling approach significantly accelerates synthesis speed. For example, generating images in 30 seconds.
- Versatility: The method successfully handles diverse tasks including style transfer, appearance transfer, style-specific text-to-image generation, controlled texture synthesis, and texture expansion to arbitrary resolutions.
- User Preference: A user study showed that the proposed method consistently outperforms alternatives by significant margins across multiple transfer tasks.
Limitations and Future Work
- Texture Expansion Saturation: For texture expansion, results can occasionally exhibit oversaturated colors. This is attributed to the AD loss not explicitly constraining data distribution consistency, relying instead on the model's understanding of the reference.
- Semantic Mismatch: When content and reference images differ significantly, the model's inherent limitations in establishing semantic correspondences can lead to incorrect matches and negatively impact the final output.
- Future Work: The authors suggest further exploration into improving texture expansion and addressing semantic mismatches in challenging cases. They also highlight the potential for combining AD loss with other conditioning mechanisms like ControlNet for even broader applications.
Summary updated