EasyDrag: Efficient Point-Based Manipulation on Diffusion Models

Xingzhong Hou, Boxiao Liu, Yi Zhang, Jihao Liu, Yu Liu, Haihang YouCVPR 2024 · OpenReview

Abstract

Generative models are gaining increasing popularity, and the demand for precisely generating images is on the rise. However, generating an image that perfectly aligns with users' expectations is extremely challenging. The shapes of objects, the poses of animals, the structures of landscapes, and more may not match the user's desires, and this applies to real images as well. This is where point-based image editing becomes essential. An excellent image editing method needs to meet the following criteria: user-friendly interaction, high performance, and good generalization capability. Due to the limitations of StyleGAN, DragGAN exhibits limited robustness across diverse scenarios, while DragDiffusion lacks user-friendliness due to the necessity of LoRA fine-tuning and masks. In this paper, we introduce a novel interactive point-based image editing framework, called EasyDrag, that leverages pretrained diffusion models to achieve high-quality editing outcomes and user-friendship. Extensive experimentation demonstrates that our approach surpasses DragDiffusion in terms of both image quality and editing precision for point-based image manipulation tasks. The code is available on https://github.com/Ace-Pegasus/EasyDrag.

Introduction

Point-based image editing—where a user clicks a "handle point" and drags it to a "target point"—has gained massive popularity thanks to tools like DragGAN. However, GAN-based methods are often limited by the capacity of the underlying model (like StyleGAN), struggling with diverse or complex real-world images.

Diffusion models offer a much broader generative space, but existing "drag" implementations for them (like DragDiffusion) have two major friction points: efficiency and usability. They often require per-image LoRA fine-tuning to preserve identity, which can take a minute or more, and they rely on manually drawn masks to keep the background stationary. EasyDrag aims to solve these issues by providing a "plug-and-play" experience that preserves identity and localizes changes automatically without requiring time-consuming fine-tuning or hand-drawn masks.

Qualitative comparison of EasyDrag against state-of-the-art methods like DragGAN and DragDiffusion.

Methodology

EasyDrag introduces three key components to make point-based manipulation efficient and stable:

  • Stable Motion Supervision: Most methods update latent features iteratively, which can lead to "feature drift" where the model loses the original object's identity. EasyDrag anchors its supervision to the feature maps of the original image. By utilizing the second and third upper layers of the U-Net denoiser, it captures high-level semantic information to guide the handle point toward the target precisely.
  • Auto Mask Generation: To avoid manual masking, EasyDrag identifies the "hot-zone" of the edit by calculating the normalized gradients of the latent code $z_t$ during the optimization process. Areas with high gradients are automatically treated as the mask, allowing the rest of the image to remain stationary.
  • Reference Guidance: To maintain identity consistency without LoRA, the framework uses a technique inspired by mutual self-attention. During the denoising process, it replaces the keys and values in the U-Net's self-attention layers with those derived from the original image's DDIM inversion path. This "injects" the original textures and details back into the edited version.
The EasyDrag pipeline featuring Stable Motion Supervision and Reference Guidance mechanisms.

Key Insights

  • LoRA is Overkill: The paper demonstrates that per-image fine-tuning isn't necessary if a strong enough guidance mechanism (Reference Guidance) and stable feature tracking are in place.
  • Gradients as a Proxy for Motion: Using the gradient of the latent code to generate a mask is an effective way to automate user intent, assuming that high-gradient areas are where the intended motion is occurring.
  • Feature Layer Selection: Not all U-Net layers are equal; the authors found that the 2nd and 3rd upper layers provide the optimal balance between spatial resolution and semantic meaning for tracking points.

Experimental Results

The primary benchmark used for evaluation is DragBench, containing 205 real-world images across various categories. EasyDrag outperforms existing state-of-the-art methods in both quality and speed:

  • Image Fidelity (IF): EasyDrag achieved a score of 0.910, surpassing DragDiffusion (0.882) and SDE-Drag (0.871).
  • Mean Distance (MD): Measuring how accurately the handle point reached the target, EasyDrag scored 32.55, significantly better than DragDiffusion's 36.71.
  • Speed: EasyDrag takes approximately 30.6 seconds per edit on an A100 GPU, compared to 69.4 seconds for DragDiffusion—a ~55% reduction in wait time.

Conclusion and Future Work

EasyDrag provides a more user-friendly and efficient approach to point-based image editing. Despite these improvements, the authors acknowledge hurdles with small-scale edits and extremely long-distance drags where the model's attention might struggle to localize changes. Future directions include leveraging diffusion models with even stronger generative capabilities, such as SDXL, to handle more dramatic structural transformations.

Summary updated

Abstract

Generative models are gaining increasing popularity, and the demand for precisely generating images is on the rise. However, generating an image that perfectly aligns with users' expectations is extremely challenging. The shapes of objects, the poses of animals, the structures of landscapes, and more may not match the user's desires, and this applies to real images as well. This is where point-based image editing becomes essential. An excellent image editing method needs to meet the following criteria: user-friendly interaction, high performance, and good generalization capability. Due to the limitations of StyleGAN, DragGAN exhibits limited robustness across diverse scenarios, while DragDiffusion lacks user-friendliness due to the necessity of LoRA fine-tuning and masks. In this paper, we introduce a novel interactive point-based image editing framework, called EasyDrag, that leverages pretrained diffusion models to achieve high-quality editing outcomes and user-friendship. Extensive experimentation demonstrates that our approach surpasses DragDiffusion in terms of both image quality and editing precision for point-based image manipulation tasks. The code will be available on https://github.com/Ace-Pegasus/EasyDrag.
BibTeX
@inproceedings{hou2024easydrag,
  author = {Xingzhong Hou and Boxiao Liu and Yi Zhang and Jihao Liu and Yu Liu and Haihang You},
  title = {{EasyDrag: Efficient Point-Based Manipulation on Diffusion Models}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  url = {https://openreview.net/forum?id=pBGb4XdCf1},
}
Figure