IterInv: Iterative Inversion for Pixel-Level T2I Models

Kai Wang (https://orcid.org/0000-0002-9605-8279), Chuanming Tang, Kai Wang, Joost van de WeijerICME 2024 · DOI

TL;DR

This paper introduces IterInv, a novel iterative inversion technique designed to overcome limitations of existing methods when editing images generated by pixel-level text-to-image diffusion models like DeepFloyd-IF. IterInv enables accurate reconstruction and controllable editing for these powerful models, which were previously incompatible with standard inversion techniques.

What Problem Does This Paper Address?

Text-to-image (T2I) diffusion models have revolutionized image generation, but controlling and editing these generated images remains a challenge. Current image editing approaches often rely on "inversion" techniques to convert a generated image back into a latent representation that can be manipulated. While DDIM inversion has been successful for latent diffusion models (LDMs) like Stable Diffusion, it struggles with pixel-level T2I models. These models, such as DeepFloyd-IF, employ a cascaded, multi-stage approach with super-resolution diffusion models. The problem arises because these super-resolution stages often condition on noisy intermediate images, which breaks the deterministic nature required for DDIM inversion, leading to poor reconstruction quality. This paper aims to bridge this gap by developing an inversion method specifically for these pixel-level, cascaded T2I models.

Methods

The core contribution is Iterative Inversion (IterInv). The authors observed that the failure of DDIM inversion in cascaded pixel-level models stems from how noisy images are concatenated as conditions in the super-resolution stages. IterInv addresses this by:

  • Leveraging Null-Text Inversion (NTI) for Stage S1: For the initial low-resolution stage, they use NTI, which is known to improve reconstruction quality by optimizing a null-text embedding.
  • Iterative Optimization for Super-Resolution Stages (S2 & S3): For the super-resolution stages, IterInv employs a novel iterative optimization process. For each diffusion timestep, it performs N steps of optimization to refine the noisy image, effectively finding a "deterministic inversion trace." This process aims to approximate the original image by minimizing the difference between the predicted denoised image and the actual noisy intermediate image.
  • Updating Latent Variance: Crucially, in stages S2 and S3, IterInv updates the latent variance at each timestep to ensure a deterministic inversion trace, promoting accurate reconstruction.
  • Compatibility with Editing Frameworks: IterInv is designed to be compatible with existing image editing methods, such as DiffEdit, allowing for text-guided editing of images generated by pixel-level T2I models.

Datasets

The paper evaluates IterInv using the ImageInversion dataset, which comprises 69 images. This dataset is a combination of images from two existing benchmarks:

  • 50 images from PnP [13]
  • 19 images from Pix2pix-zero [26]

The authors state that these images are resized to 1024x1024 for evaluation. No new datasets were introduced.

Key Insights and Ideas

  • The "Concatenation Conditioning" Problem: The paper clearly identifies and articulates the root cause of inversion failure in cascaded pixel-level diffusion models: the way noisy intermediate images are used as conditions in super-resolution stages breaks the deterministic inversion process.
  • Iterative Refinement for Robust Inversion: The core idea of iteratively refining the noisy image at each timestep in the super-resolution stages is a clever way to overcome the limitations of standard deterministic inversion. It allows the model to "learn" the correct path back to the original image despite the complex conditioning.
  • Pixel-Level Editing is Now Feasible: This work demonstrates that high-quality image editing is achievable with pixel-level T2I models, which were previously difficult to edit due to inversion challenges. This opens up new avenues for research and application in this domain.

Key Results

  • Superior Reconstruction Quality: Table I shows that IterInv significantly outperforms DDIM inversion across various stages (s3, s2+s3, s1+s2+s3) and even the SDXL autoencoder in terms of MSE, LPIPS, SSIM, and PSNR. For instance, IterInv achieves an MSE of 0.000129, which is orders of magnitude lower than DDIM (s1,s2,s3) at 0.079924.
  • High-Quality Image Editing: Figure 2 visually demonstrates the effectiveness of IterInv when combined with DiffEdit. The edited images show accurate content manipulation while preserving image quality and details, a feat not easily achieved with previous methods on these models.
  • Parameter Robustness: An ablation study (Table I) shows that IterInv is robust to variations in the classifier-free guidance scale (w₁) for stage S1, maintaining similar reconstruction performance.

Limitations and Future Work

  • Limited to DeepFloyd-IF: The current implementation and evaluation are specifically for the DeepFloyd-IF model. The authors acknowledge that the applicability to other pixel-level T2I models might require adaptation.
  • Tested with One Editing Method: The paper primarily demonstrates IterInv's compatibility with the DiffEdit framework. Its effectiveness with other image editing methodologies remains to be explored.
  • Future Work: The authors suggest exploring the compatibility and effectiveness of IterInv with a wider range of editing techniques to broaden its applicability.

Summary updated

BibTeX
@inproceedings{httpsorcidorg00000002960582792024iterinv,
  author = {Kai Wang (https://orcid.org/0000-0002-9605-8279) and Chuanming Tang and Kai Wang and Joost van de Weijer},
  title = {{IterInv: Iterative Inversion for Pixel-Level T2I Models}},
  booktitle = {ICME},
  year = {2024},
  doi = {10.1109/icme57554.2024.10687547},
}
Figure