Abstract
Subject-driven image editing requires the precise insertion of a specific subject from a reference image into a new context while maintaining the subject's unique identity and the model's text-driven controllability. Current methods typically encode reference subjects into compressed embeddings, leading to a "bottleneck" that loses fine-grained details. We introduce PhD (Paste, Inpaint, and Harmonize via Denoising), a framework that bypasses this bottleneck by directly pasting subject pixels into the target image. A dedicated harmonization module then guides a frozen diffusion model to blend the subject seamlessly. Experimental results demonstrate that PhD significantly outperforms existing baselines in both identity preservation and image realism.
Introduction
The ability to take a specific subject—such as a user's own pet or a unique heirloom—and place it into a novel scene is a core challenge in generative AI. While large-scale diffusion models like Stable Diffusion excel at generating generic categories, they often fail to capture the specific "identity" of a provided exemplar.
Previous state-of-the-art approaches, such as Paint-by-Example (PbE), rely on CLIP embeddings to represent the reference subject. However, CLIP is designed for high-level semantic understanding and often discards low-level textures and specific markings essential for identity. Furthermore, replacing text features with image features often degrades the model's ability to follow complex text prompts. The PhD framework addresses this trade-off by prioritizing pixel-level fidelity and leveraging the generative power of frozen diffusion models.
Methods
The PhD framework operates through a two-stage pipeline designed to maintain 100% of the subject's visual details while ensuring environmental consistency.
1. The Paste Step
Rather than describing the subject to the model, PhD utilizes off-the-shelf segmentation tools (e.g., SAM or U²Net) to extract the subject from the exemplar image. This subject is then directly pasted onto the target background within a user-defined mask. This "hard" insertion ensures that all textures, colors, and patterns are preserved perfectly from the outset.
2. Inpaint and Harmonize via Denoising
To resolve the visual artifacts of a simple paste (e.g., mismatched lighting and sharp edges), PhD introduces a trainable harmonization module. Inspired by ControlNet, this module processes the pasted image and the mask to generate guidance signals for a frozen pre-trained diffusion model. By keeping the main model frozen, PhD preserves the original model's creative versatility and text-following capabilities. The harmonization module is trained using a self-supervised reconstruction task on 130k images from the Open-Images v7 dataset.
Results and Discussion
PhD was evaluated against the COCO Exemplar-based Image Editing Benchmark. Quantitatively, PhD achieved a FID_scene of 1.71, outperforming PbE (2.07), and a superior CLIP_I identity score of 78.32.
Key Insights
- Identity Preservation: By using actual pixels instead of embeddings, PhD avoids the "hallucination" of generic subjects common in previous methods.
- Harmonization as Guidance: Treating harmonization as a guidance signal allows the diffusion process to naturally adjust geometry and lighting, making the subject appear truly integrated into the scene.
- Controllability: Because the core diffusion model remains untouched, users can still apply global style transfers (e.g., "pencil drawing") while keeping the subject's identity intact.
In a user study with 25 participants, PhD's outputs were preferred 48.9% of the time, significantly higher than the 21.6% preference for PbE.
Conclusion
The PhD framework demonstrates that direct pixel manipulation, when combined with sophisticated denoising-based harmonization, provides a superior path for subject-driven image editing. While currently limited by its 2D nature when generating unseen perspectives, future work integrating 3D priors could further enhance geometric consistency.
Summary updated