VersaFusion: A Versatile Diffusion-Based Framework for Fine-Grained Image Editing and Enhancement

Haocun Ye, Xinlong Jiang, Chenlong Gao, Bingyu Wang, Wuliang Huang, Yiqiang Chen

Introduction

Precise, pixel-level control remains a significant challenge in Text-to-Image (T2I) diffusion models. While models like Stable Diffusion excel at generating images from text, they often lack the granularity required for specific edits, such as adjusting the position of an object or changing a facial expression. Existing methods like DragDiffusion attempt point-based manipulation but frequently suffer from a "fidelity-controllability" trade-off, either failing to preserve the original image's identity or requiring computationally expensive per-image fine-tuning.

VersaFusion addresses these limitations with a tuning-free, dual-branch framework designed for high-fidelity editing tasks, including "click-and-drag" manipulations, style transfer, and outpainting.

Demonstration of VersaFusion's capability for contextual canvas extension (outpainting).

Methodology

The core of VersaFusion is a dual-branch architecture that decouples image reconstruction from the editing process:

  1. Guidance Branch: This branch focuses solely on reconstructing the original input image with high fidelity, acting as a reference library of features.
  2. Generation Branch: This branch performs the actual editing. To maintain structural consistency, it "borrows" information from the Guidance Branch via Key-Value (KV) replacement in the self-attention layers.

Technical Innovations

  • Proximal Negative-Prompt Inversion (PNI): Standard inversion techniques often introduce drift, leading to blurry results. PNI incorporates a regularization term (using $L_1$ or $L_0$ norms) to align noise during inversion, ensuring sharper reconstructions and edits. It is approximately 19x faster than standard Null-text inversion.
  • Multi-Modal Prompting: Beyond text, VersaFusion utilizes an Image Prompt Encoder (QFormer) to incorporate reference images for tasks like style transfer or specific attribute modification.
  • Score-Based Guidance: User-defined "drag" points are treated as gradient optimization targets, nudging latent pixels toward the desired locations during each denoising step.

Experimental Results

VersaFusion was evaluated on face manipulation tasks using the LAION dataset and benchmarked against state-of-the-art methods using FID (Fréchet Inception Distance) scores and landmark accuracy (MSE).

Qualitative and Quantitative Performance

VersaFusion demonstrates a superior ability to "hallucinate" missing details, such as the interior of an eye when dragged from a closed to an open state. Quantitatively, it achieved a leading FID of 32.98 and a landmark accuracy of 11.45 MSE, outperforming both DragDiffusion and DragonDiffusion.

Comparison of VersaFusion (VDiff) against DragDiffusion and DragonDiffusion for point-based eye manipulation.

Efficiency

A major advantage of VersaFusion is its speed. By avoiding per-image fine-tuning (such as LoRA), the preparation phase takes approximately 5.23 seconds, compared to over 60 seconds for older GAN-based or diffusion-based methods. This efficiency makes it highly viable for real-time interactive applications.

Conclusion and Future Work

VersaFusion provides a robust, plug-and-play solution for interactive image editing without sacrificing fidelity. While it excels at attribute and position manipulation, it still faces challenges in "high-imagination" scenarios where entirely new complex objects must be generated from scratch. Future research will focus on integrating Large Language Models (LLMs) for better instruction following and extending the framework to video and 3D object manipulation.

Summary updated

Abstract

Text-to-image (T2I) diffusion models have achieved remarkable progress in generating realistic images from textual descriptions. However, ensuring consistent high-quality image generation with complete backgrounds, object appearance, and optimal texture rendering remains challenging. This paper presents a novel fine-grained pixel-level image editing method based on pre-trained diffusion models. The proposed dual-branch architecture, consisting of Guidance and Generation branches, employs U-Net Denoisers and Self-Attention mechanisms. An improved DDIM-like inversion method obtains the latent representation, followed by multiple denoising steps. Cross-branch interactions, such as KV Replacement, Classifier Guidance, and Feature Correspondence, enable precise control while preserving image fidelity. The iterative refinement and reconstruction process facilitates finegrained editing control, supporting attribute modification, image outpainting, style transfer, and face synthesis with Clickand-Drag style editing using masks. Experimental results demonstrate the effectiveness of the proposed approach in enhancing the quality and controllability of T2I-generated images, surpassing existing methods while maintaining attractive computational complexity for practical real-world applications.
BibTeX
@inproceedings{ye2025versafusion,
  author = {Haocun Ye and Xinlong Jiang and Chenlong Gao and Bingyu Wang and Wuliang Huang and Yiqiang Chen},
  title = {{VersaFusion: A Versatile Diffusion-Based Framework for Fine-Grained Image Editing and Enhancement}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i9.33023},
}
Figure