LASPA: Latent Spatial Alignment for Fast Training-free Single Image Editing

Yazeed Alharbi, Peter WonkaarXiv 2024 · DOI · arXiv

Abstract

LASPA (LAtent SPatial Alignment) is a novel, training-free framework designed for high-quality text-based editing of real images. By aligning the spatial latents of a diffusion model with the reference image's features during the denoising process, LASPA achieves state-of-the-art results in under 6 seconds. Unlike existing methods, it requires no model finetuning or complex optimization, making it highly efficient for both computation and storage.

Introduction

Text-to-image diffusion models have revolutionized content creation, yet editing existing real images remains computationally expensive. Current state-of-the-art techniques typically rely on inverting images into the latent space or finetuning model parameters to capture specific object identities. These processes are often slow, taking several minutes per image, and require significant storage for per-image model weights.

LASPA addresses these challenges by introducing an inference-only guidance mechanism. It preserves the original image's identity and structure while faithfully following new textual instructions, all without modifying the underlying model weights.

Methodology: Latent Spatial Alignment

The core of the LASPA framework is the guidance of spatial latents throughout the reverse diffusion process. The authors argue that spatial latents are inherently superior to textual tokens for encoding fine-grained visual details, bypassing the "information bottleneck" often found in text-based inversion.

Alignment Strategies

The researchers evaluated three primary alignment methods: Input Alignment, epsilon-theta Alignment, and x0 Prediction Alignment. The study concluded that aligning the x0 prediction (Equation 9 in the paper) offers the optimal balance. Early in the diffusion process, the model is steered toward the original image's structure. As denoising progresses, this alignment is gradually relaxed, providing the model with the "freedom" to incorporate textual edits.

Style and content edits on the 'Girl with a Pearl Earring'. LASPA maintains high fidelity to the original composition while accurately reflecting complex prompts like 'Cthulhu art'.

Semantic Latent Mixing

To further enhance precision, LASPA utilizes Semantic Latent Mixing. This technique leverages the model's internal attention maps as masks, allowing for localized edits (e.g., background replacement) while keeping the primary subject unchanged. Notably, LASPA generates these clean attention maps in a single forward pass, whereas competing methods often require extensive optimization.

Results and Performance

LASPA was evaluated using the TEdBench benchmark and various base models, including Stable Diffusion (v1.4, v2.1), Wuerstchen, and Latent Consistency Models (LCMs).

Speed and Efficiency

LASPA performs edits in less than 6 seconds, a significant improvement over methods like Imagic (~8 minutes) and SINE (~15 minutes). Furthermore, it requires 0GB of additional storage, as it operates on frozen, pre-trained weights.

Quality and User Preference

In a user study involving over 1,200 responses, LASPA was preferred 71% of the time over SDEdit and 62% of the time over SINE. Quantitative metrics (LPIPS for preservation and CLIP similarity for editing strength) further validate that LASPA maintains high image fidelity without sacrificing the quality of the requested edits.

LASPA enables a wide range of edits, including object replacement, attribute changes, and style transfer, all while preserving the original image's structure.

Conclusion

LASPA provides a fast, training-free solution for single-image editing. While it excels at maintaining structure and style, it faces challenges with large-scale geometric transformations (e.g., significant pose changes). Future research will explore combining spatial alignment with advanced control mechanisms to handle complex structural overhauls while maintaining the framework's efficiency.

Summary updated

Abstract

We present a novel, training-free approach for textual editing of real images using diffusion models. Unlike prior methods that rely on computationally expensive finetuning, our approach leverages LAtent SPatial Alignment (LASPA) to efficiently preserve image details. We demonstrate how the diffusion process is amenable to spatial guidance using a reference image, leading to semantically coherent edits. This eliminates the need for complex optimization and costly model finetuning, resulting in significantly faster editing compared to previous methods. Additionally, our method avoids the storage requirements associated with large finetuned models. These advantages make our approach particularly well-suited for editing on mobile devices and applications demanding rapid response times. While simple and fast, our method achieves 62-71\% preference in a user-study and significantly better model-based editing strength and image preservation scores.
BibTeX
@misc{alharbi2024laspa,
  author = {Yazeed Alharbi and Peter Wonka},
  title = {{LASPA: Latent Spatial Alignment for Fast Training-free Single Image Editing}},
  year = {2024},
  eprint = {2403.12585},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2403.12585},
}
Figure