Paste and Harmonize via Denoising: Subject-Driven Image Editing with Frozen Pre-Trained Diffusion Model

Xin Zhang, Jiaxian Guo, Paul Yoo, Yutaka Matsuo, Yusuke Iwasawa

Abstract

Subject-driven image editing requires the precise insertion of a specific subject from a reference image into a new context while maintaining the subject's unique identity and the model's text-driven controllability. Current methods typically encode reference subjects into compressed embeddings, leading to a "bottleneck" that loses fine-grained details. We introduce PhD (Paste, Inpaint, and Harmonize via Denoising), a framework that bypasses this bottleneck by directly pasting subject pixels into the target image. A dedicated harmonization module then guides a frozen diffusion model to blend the subject seamlessly. Experimental results demonstrate that PhD significantly outperforms existing baselines in both identity preservation and image realism.

Introduction

The ability to take a specific subject—such as a user's own pet or a unique heirloom—and place it into a novel scene is a core challenge in generative AI. While large-scale diffusion models like Stable Diffusion excel at generating generic categories, they often fail to capture the specific "identity" of a provided exemplar.

Previous state-of-the-art approaches, such as Paint-by-Example (PbE), rely on CLIP embeddings to represent the reference subject. However, CLIP is designed for high-level semantic understanding and often discards low-level textures and specific markings essential for identity. Furthermore, replacing text features with image features often degrades the model's ability to follow complex text prompts. The PhD framework addresses this trade-off by prioritizing pixel-level fidelity and leveraging the generative power of frozen diffusion models.

Methods

The PhD framework operates through a two-stage pipeline designed to maintain 100% of the subject's visual details while ensuring environmental consistency.

Overview of the PhD framework: Subject pixels are extracted and pasted into the target region, followed by a harmonization step guided by a frozen diffusion model.

1. The Paste Step

Rather than describing the subject to the model, PhD utilizes off-the-shelf segmentation tools (e.g., SAM or U²Net) to extract the subject from the exemplar image. This subject is then directly pasted onto the target background within a user-defined mask. This "hard" insertion ensures that all textures, colors, and patterns are preserved perfectly from the outset.

2. Inpaint and Harmonize via Denoising

To resolve the visual artifacts of a simple paste (e.g., mismatched lighting and sharp edges), PhD introduces a trainable harmonization module. Inspired by ControlNet, this module processes the pasted image and the mask to generate guidance signals for a frozen pre-trained diffusion model. By keeping the main model frozen, PhD preserves the original model's creative versatility and text-following capabilities. The harmonization module is trained using a self-supervised reconstruction task on 130k images from the Open-Images v7 dataset.

Results and Discussion

PhD was evaluated against the COCO Exemplar-based Image Editing Benchmark. Quantitatively, PhD achieved a FID_scene of 1.71, outperforming PbE (2.07), and a superior CLIP_I identity score of 78.32.

Qualitative comparison between PhD and baselines. PhD maintains intricate subject details (e.g., clock faces, vase patterns) that other methods lose.

Key Insights

  • Identity Preservation: By using actual pixels instead of embeddings, PhD avoids the "hallucination" of generic subjects common in previous methods.
  • Harmonization as Guidance: Treating harmonization as a guidance signal allows the diffusion process to naturally adjust geometry and lighting, making the subject appear truly integrated into the scene.
  • Controllability: Because the core diffusion model remains untouched, users can still apply global style transfers (e.g., "pencil drawing") while keeping the subject's identity intact.

In a user study with 25 participants, PhD's outputs were preferred 48.9% of the time, significantly higher than the 21.6% preference for PbE.

Conclusion

The PhD framework demonstrates that direct pixel manipulation, when combined with sophisticated denoising-based harmonization, provides a superior path for subject-driven image editing. While currently limited by its 2D nature when generating unseen perspectives, future work integrating 3D priors could further enhance geometric consistency.

Summary updated

Abstract

Text-to-Image generative models have shown a remarkable ability to produce high-quality images. However, existing methods still face difficulties in exemplar-guided image editing without destroying the given objects’ identity in the exemplar image. To address this problem, we propose a new framework called Paste and Harmonize via Denoising, which leverages pre-trained diffusion models to facilitate the text-driven transfer of objects from an exemplar image to the edited image while preserving their appearance and characteristics. The framework consists of two main steps: paste and harmonize via denoising. In the paste step, an off-the-shelf text-driven model is utilized to localize the objects in the exemplar image. The editing task is naturally transformed into an image harmonization task by pasting the object patches into the edited image. In the harmonize via denoising step, we introduce an image harmonization module based on pre-trained diffusion models to blend the inserted object with the target image, producing a coherent and realistic image without compromising synthesis quality and preserving the text-driven style transfer editing ability. In the experiments, the qualitative comparisons with baselines demonstrate that our method achieves impressive performance in exemplar-based image editing on both training and in-the-wild images with high fidelity. More qualitative and quantitative results can be found at our website.
BibTeX
@inproceedings{zhang2024paste,
  author = {Xin Zhang and Jiaxian Guo and Paul Yoo and Yutaka Matsuo and Yusuke Iwasawa},
  title = {{Paste and Harmonize via Denoising: Subject-Driven Image Editing with Frozen Pre-Trained Diffusion Model}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  doi = {10.1109/icassp48485.2024.10448510},
}

Loading related content...

Figure