Abstract
This paper presents EntityErasure, a novel diffusion-based inpainting method that can effectively erase entities without inducing unwanted sundries. To this end, we propose to address this problem by dividing it into amodal entity segmentation and completion, such that the region to inpaint takes only entities in the non-inpainting area as reference, avoiding the possibility to generate unpredictable sundries. Moreover, we develop two entity segmentation based metrics for quantitatively assessing the performance of object erasure, which are shown be more effective than existing metrics. Experimental results demonstrate that our approach outperforms other state-of-the-art object erasure methods.
Introduction
A persistent challenge in image editing is the clean removal of objects. While modern generative models like Stable Diffusion are highly capable of filling missing regions, they are often "too creative," frequently hallucinating unwanted artifacts or new objects—termed "sundries"—in the erased area. For instance, attempting to remove a person from a park might result in the model generating a bag or a bench that doesn't fit the existing scene geometry. Existing solutions relying on text prompts often fail in complex environments because they lack a structural understanding of the background entities hidden behind the target object.
Methodology
The proposed EntityErasure framework reframes object erasure as a multi-entity amodal completion task. The pipeline consists of two primary stages:
- Amodal Entity Segmentation Model (AESM): This stage predicts the full "amodal" masks of all entities in the scene. Unlike standard segmentation, AESM estimates the complete shape of background objects, including the portions occluded by the entity to be erased.
- Amodal Entity Completion Model (AECM): This generative engine utilizes the amodal masks to guide the inpainting process. Its core component is Entity Attention (EA), which constrains the denoising process. By forcing pixels in the erased region to attend only to features from relevant background entities, the model is physically prevented from introducing random features that could form sundries.
The framework also employs an enhanced version of Classifier-Free Guidance (CFG) to prioritize background context over unconstrained generation.
Experimental Setup
The model was trained on 1.75 million images from the Open Images V7 dataset, using pseudo-labels from a pre-trained segmentation model for amodal masks. Evaluation was conducted on the Entity Segmentation test set (700 samples) and COCO val2017 (4,000 samples).
To better quantify the "cleanliness" of erasure, the authors introduced two new metrics:
- Mean Sundries Num (MSN): The average count of unintended objects created.
- Mean Area Ratio of Sundries (MARS): The relative size of unwanted artifacts within the erased region.
Results and Discussion
EntityErasure demonstrates a significant reduction in artifacts compared to state-of-the-art methods like PowerPaint, CLIPAway, and MAT.
Quantitatively, on the EntitySeg dataset, EntityErasure achieved an MSN of 0.122, a substantial improvement over PowerPaint (0.554) and CLIPAway (0.679). The model also scored higher on the ReMOVE metric, indicating superior contextual integration. Inference is efficient, taking approximately 2.98 seconds on an NVIDIA RTX 4090.
Conclusion
EntityErasure provides a robust solution for clean object removal by treating erasure as entity completion. While the method is highly effective, it remains dependent on the accuracy of the initial amodal segmentation. Future work will focus on improving amodal prediction for highly occluded scenes and integrating explicit shadow handling to further enhance the realism of the final output.
Summary updated