AID: Attention Interpolation of Text-to-Image Diffusion

Qiyuan He, Jinghao Wang, Ziwei Liu, Angela Yao

Introduction

Generative models, particularly text-to-image diffusion models like Stable Diffusion, have revolutionized the creation of high-fidelity imagery from natural language. However, a persistent challenge remains: achieving smooth, semantically coherent transitions between two distinct concepts—for instance, morphing a "truck" into a "cat." While users can easily generate endpoints, the path between them often suffers from structural collapse or abrupt visual "jumps."

This paper introduces Attention Interpolation via Diffusion (AID), a training-free technique designed to enable high-fidelity visual transitions. By manipulating the internal attention mechanisms of the diffusion process and employing a non-linear sampling schedule, AID significantly outperforms standard interpolation methods in both consistency and image quality.

The Limitations of Text Embedding Interpolation

The conventional approach to transitioning between prompts is Text Embedding Interpolation (TEI). In TEI, the mathematical vectors representing two prompts are averaged using weighted coefficients. While intuitive, this method frequently fails in practice. Intermediate frames often result in "messy sketches" or incoherent artifacts.

The authors identify a fundamental structural reason for this failure: TEI primarily influences the cross-attention layers, which map text tokens to specific pixel regions. However, it largely ignores the self-attention layers, which are responsible for the overall spatial layout and geometric structure of the image. Without explicit control over self-attention, the model loses thematic consistency during the morphing process, leading to the chaotic transitions observed in standard methods.

The AID Framework

AID operates as a post-hoc modification to the diffusion process, requiring no additional training or fine-tuning. The framework consists of three core innovations:

1. Fused Interpolated Attention

Instead of relying solely on interpolated text embeddings, AID modifies the attention mechanism itself. It introduces a "fused" approach where the keys and values of the attention layers are derived from a combination of the source, target, and interpolated states. This ensures the model maintains the structural integrity of the endpoints while generating intermediate frames. The authors propose two variants:

  • AID-I (Inner): Interpolates keys and values before the attention operation occurs.
  • AID-O (Outer): Interpolates the outputs of the attention layers.

2. Beta-Distribution Sampling

Visual transitions in latent space are rarely linear. A sequence might retain the characteristics of the source image for 80% of the interpolation path before suddenly snapping to the target. To combat this "jump" phenomenon, AID utilizes a Beta distribution to select interpolation coefficients. This acts as a "visual gear shift," slowing down the sampling process during critical transition zones to ensure a much smoother morphing effect.

3. Prompt-guided AID (PAID)

For more complex transitions, the authors introduce PAID. This variant allows for a third "guidance prompt" to define the interpolation path. For example, when transitioning from a "dog" to a "car," a user might provide the guidance prompt "a dog driving a car." This ensures that the intermediate steps are not just mathematical blends but conceptually grounded images.

Experimental Insights

The researchers evaluated AID using the CIFAR-10 and LAION-Aesthetics datasets. Their analysis yielded several key insights:

  • Self-Attention and Layout: Diagnostic tests confirmed that self-attention has a dominant influence on spatial layout compared to cross-attention. By controlling self-attention, AID maintains structural stability that TEI lacks.
  • Quantitative Performance: On CIFAR-10, AID-O improved consistency by approximately 20% and reduced the Fréchet Inception Distance (FID)—a measure of image quality—by over 50% compared to the TEI baseline.
  • Human Preference: Qualitative studies showed a strong preference for AID. Users favored AID-I for its exceptional smoothness and AID-O for its ability to preserve the distinct identities of the source and target images.

Applications and Future Work

Beyond simple morphing, AID enables advanced applications such as compositional generation (e.g., ensuring two distinct concepts appear together correctly) and precise image editing. Because it is training-free, it can be integrated into existing architectures like Stable Diffusion 1.4 or SDXL without additional computational overhead.

While AID is limited by the capabilities of the underlying base model, it represents a significant step forward in controllable generative AI. Future research may extend these attention-manipulation techniques to video diffusion models, where maintaining temporal consistency across frames is a critical requirement. Additionally, the authors note the importance of developing generative watermarking to mitigate the potential misuse of seamless image morphing in creating deepfakes.

Summary updated

BibTeX
@inproceedings{he2024aid,
  author = {Qiyuan He and Jinghao Wang and Ziwei Liu and Angela Yao},
  title = {{AID: Attention Interpolation of Text-to-Image Diffusion}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=Nb5xlelV0C},
}
Figure