Introduction
Generating faithful variations of real-world images remains a significant challenge for text-to-image diffusion models. While models like Stable Diffusion excel at creating images from scratch, they often fail to preserve the specific textures, color tones, and lighting of a user-provided exemplar. This paper introduces RIVAL (Real-world Image Variation by ALignment), a training-free inference pipeline designed to bridge the "distribution gap" between real-image latents and standard Gaussian noise.
The Distribution Gap Problem
Standard diffusion models are trained to start from pure Gaussian noise. However, when a real image is "inverted" into noise to serve as a starting point for variations, the resulting latents are often biased. This misalignment accumulates during the denoising process, causing the generated image to drift away from the original's unique characteristics. RIVAL addresses this by aligning the generation process with the source image's inversion chain.
Methodology: Dual-Chain Alignment
RIVAL operates as a "plug-and-play" strategy requiring no fine-tuning. It utilizes two parallel diffusion chains: an inversion chain (reconstructing the original) and a generation chain (creating the variation). Two key mechanisms ensure consistency:
- Cross-Image Self-Attention Injection: The generation chain's self-attention layers are modified to incorporate Key ($K$) and Value ($V$) features from the inversion chain. This allows the model to borrow structural and stylistic cues directly from the real image.
- Step-wise Distribution Normalization: To counteract latent distribution drift, RIVAL employs an adaptive normalization technique. In the early denoising stages, it aligns the noise predictions of the generation chain with those of the inversion chain, maintaining the "soul" of the exemplar.
Experimental Evaluation
The authors evaluated RIVAL using images from the internet and the DreamBooth dataset, with prompts generated via BLIP-2. Performance was measured using CLIP-score (alignment), Palette Distance (color similarity), and Kernel Inception Distance (KID) for perceptual quality.
Key Results and Insights
RIVAL demonstrates superior performance over competitive baselines such as ELITE and DALL-E 2.
- Perceptual Quality: RIVAL achieved a KID score of 13.2, outperforming UnCLIP (13.5) and ELITE (25.7).
- Color Consistency: The method led in Palette Distance metrics, proving its ability to preserve complex lighting and color schemes that other models often wash out.
- Efficiency: Variations are generated in approximately 8 seconds on an NVIDIA RTX 4090, making it highly practical for real-world applications.
Conclusion
RIVAL proves that alignment throughout the denoising process—not just at the start—is crucial for real-world image variation. By treating the inversion chain as a reference for self-attention and distribution normalization, RIVAL achieves high-fidelity results without the need for costly optimization or training. Future work will look to reduce dependence on text prompts and improve inversion for highly complex scenes.
Summary updated