Real-World Image Variation by Aligning Diffusion Inversion Chain

Yuechen Zhang, Jinbo Xing, Eric Lo, Jiaya Jia

Introduction

Generating faithful variations of real-world images remains a significant challenge for text-to-image diffusion models. While models like Stable Diffusion excel at creating images from scratch, they often fail to preserve the specific textures, color tones, and lighting of a user-provided exemplar. This paper introduces RIVAL (Real-world Image Variation by ALignment), a training-free inference pipeline designed to bridge the "distribution gap" between real-image latents and standard Gaussian noise.

The Distribution Gap Problem

Standard diffusion models are trained to start from pure Gaussian noise. However, when a real image is "inverted" into noise to serve as a starting point for variations, the resulting latents are often biased. This misalignment accumulates during the denoising process, causing the generated image to drift away from the original's unique characteristics. RIVAL addresses this by aligning the generation process with the source image's inversion chain.

Methodology: Dual-Chain Alignment

RIVAL operates as a "plug-and-play" strategy requiring no fine-tuning. It utilizes two parallel diffusion chains: an inversion chain (reconstructing the original) and a generation chain (creating the variation). Two key mechanisms ensure consistency:

  1. Cross-Image Self-Attention Injection: The generation chain's self-attention layers are modified to incorporate Key ($K$) and Value ($V$) features from the inversion chain. This allows the model to borrow structural and stylistic cues directly from the real image.
  2. Step-wise Distribution Normalization: To counteract latent distribution drift, RIVAL employs an adaptive normalization technique. In the early denoising stages, it aligns the noise predictions of the generation chain with those of the inversion chain, maintaining the "soul" of the exemplar.
RIVAL produces high-quality variations across diverse categories like architecture, objects, and animals while maintaining stylistic consistency.

Experimental Evaluation

The authors evaluated RIVAL using images from the internet and the DreamBooth dataset, with prompts generated via BLIP-2. Performance was measured using CLIP-score (alignment), Palette Distance (color similarity), and Kernel Inception Distance (KID) for perceptual quality.

Key Results and Insights

RIVAL demonstrates superior performance over competitive baselines such as ELITE and DALL-E 2.

  • Perceptual Quality: RIVAL achieved a KID score of 13.2, outperforming UnCLIP (13.5) and ELITE (25.7).
  • Color Consistency: The method led in Palette Distance metrics, proving its ability to preserve complex lighting and color schemes that other models often wash out.
  • Efficiency: Variations are generated in approximately 8 seconds on an NVIDIA RTX 4090, making it highly practical for real-world applications.
A variation of a classic portrait demonstrating RIVAL's ability to maintain semantic identity and lighting.

Conclusion

RIVAL proves that alignment throughout the denoising process—not just at the start—is crucial for real-world image variation. By treating the inversion chain as a reference for self-attention and distribution normalization, RIVAL achieves high-fidelity results without the need for costly optimization or training. Future work will look to reduce dependence on text prompts and improve inversion for highly complex scenes.

Summary updated

Abstract

Recent diffusion model advancements have enabled high-fidelity images to be generated using text prompts. However, a domain gap exists between generated images and real-world images, which poses a challenge in generating high-quality variations of real-world images. Our investigation uncovers that this domain gap originates from a latents' distribution gap in different diffusion processes. To address this issue, we propose a novel inference pipeline called Real-world Image Variation by ALignment (RIVAL) that utilizes diffusion models to generate image variations from a single image exemplar. Our pipeline enhances the generation quality of image variations by aligning the image generation process to the source image's inversion chain. Specifically, we demonstrate that step-wise latent distribution alignment is essential for generating high-quality variations. To attain this, we design a cross-image self-attention injection for feature interaction and a step-wise distribution normalization to align the latent features. Incorporating these alignment processes into a diffusion model allows RIVAL to generate high-quality image variations without further parameter optimization. Our experimental results demonstrate that our proposed approach outperforms existing methods concerning semantic similarity and perceptual quality. This generalized inference pipeline can be easily applied to other diffusion-based generation tasks, such as image-conditioned text-to-image generation and stylization. Project page: https://rival-diff.github.io
BibTeX
@inproceedings{zhang2023real,
  author = {Yuechen ZHANG and Jinbo Xing and Eric Lo and Jiaya Jia},
  title = {{Real-World Image Variation by Aligning Diffusion Inversion Chain}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=u6Ibs4hTJH},
}
Figure