Referring Image Editing: Object-Level Image Editing via Referring Expressions

Chang Liu, Xiangtai Li, Henghui DingCVPR 2024 · OpenReview

Abstract

Significant advancements have been made in image editing with the recent advance of the Diffusion model. However, most current methods primarily focus on global or subject-level modifications and often face limitations when editing specific objects when other objects coexist in the scene, given solely textual prompts. In response, we introduce Referring Image Editing (RIE), an object-level generative task that enables identifying and editing specific source objects using text prompts. We propose ReferDiffusion, a framework that disentangles input prompts into multiple embeddings and employs a mixed-supervised multi-stage training strategy. We also introduce the RefCOCO-Edit dataset for training and evaluation. Experiments demonstrate our approach's effectiveness in identifying and editing target objects compared to conventional methods.

Introduction: The Challenge of Precision in Image Editing

While AI image editing has exploded with models like Stable Diffusion and InstructPix2Pix, most current tools struggle with precision in crowded scenes. Existing methods generally fall into two categories:

  1. Global/Subject-level Editing: These models respond to prompts like "change the dog to a cat." However, if an image contains multiple dogs, these models often lack the discriminatory ability to pick the specific target, leading to unintended changes across the entire scene.
  2. Region-based Editing: These allow for precise edits but require users to manually provide a mask or bounding box, which is less intuitive than natural language.

Referring Image Editing (RIE) bridges this gap, enabling a "point-and-edit" experience using only text, even when multiple similar objects are present.

Comparison of General, Region-based, and Referring Image Editing tasks.

Methodology: The ReferDiffusion Framework

To address the dual requirements of RIE—discrimination (identifying the object) and generation (editing it)—we developed ReferDiffusion. The architecture moves beyond treating text prompts as monolithic inputs by disentangling them into functional components.

Region-Embedder Module (REM)

The REM extracts three distinct embeddings from the input:

  • Vision Embedding: Captures context from the source image.
  • Referring Embedding: Encodes the "where" (spatial instructions).
  • Edition Embedding: Encodes the "what" (the desired change).

The module also predicts an internal object mask to guide the model's attention during the diffusion process.

Region-Control Loss (RCL)

To maintain background integrity, the Region-Control Loss penalizes the model for changing pixels outside the identified target area. This ensures that the rest of the image remains static while the target object is modified, solving a common issue where general-purpose models subtly change the entire image's lighting or texture.

Mixed-Supervised Training

High-quality "before and after" editing data is scarce. ReferDiffusion uses a two-stage training approach:

  1. Self-supervised Stage: The model is trained on existing segmentation datasets (RefCOCO/+/g). It learns to "paint" new objects over masked areas, teaching it localization and generation fundamentals.
  2. Editing Stage: The model is refined using real editing data to master complex transformations.

Evaluation and Results

We introduced RefCOCO-Edit, a benchmark dataset derived from 400 images in the RefCOCO dataset. It includes manually created editing prompts, source masks, and reference edited images for rigorous evaluation.

Quantitative Performance

ReferDiffusion was compared against state-of-the-art models including InstructPix2Pix, InstructDiffusion, and MagicBrush. It achieved the best Background Consistency (L2bg) score (0.028) and the highest Foreground CLIP-Similarity, indicating superior alignment with the text prompt without degrading the background.

Qualitative and User Study

In head-to-head comparisons, ReferDiffusion consistently distinguished between similar objects (e.g., "the left pizza slice" vs. "the right pizza slice"). A user study with 50 participants ranked ReferDiffusion significantly higher in consistency (3.63/5) compared to InstructDiffusion (3.15/5).

Discussion and Limitations

The core insight of this work is that disentanglement is key. By separating the referring part of a prompt from the editing part, ReferDiffusion avoids the "identity confusion" common in global models. Furthermore, results suggest that localization must precede editing; integrating segmentation-like tasks directly into the diffusion process is essential for object-level control.

However, limitations remain. ReferDiffusion's object identification is not yet as robust as specialized non-generative Referring Expression Segmentation (RES) models. Additionally, the final image quality is tied to the underlying base model (Stable Diffusion v1.5). Future work will explore more powerful vision-language backbones and scaling the dataset to handle more complex spatial reasoning and intricate object interactions.

Summary updated

Abstract

Significant advancements have been made in image editing with the recent advance of the Diffusion model. How-ever, most of the current methods primarily focus on global or subject-level modifications, and often face limitations when it comes to editing specific objects when there are other objects coexisting in the scene, given solely textual prompts. In response to this challenge, we introduce an object-level generative task called Referring Image Editing (RIE), which enables the identification and editing of specific source objects in an image using text prompts. To tackle this task effectively, we propose a tailoredframework called ReferDiffusion. It aims to disentangle input prompts into multiple embeddings and employs a mixed-supervised multi-stage training strategy. To facilitate further research in this domain, we introduce the Ref COCO-Edit dataset, comprising images, editing prompts, source object segmen-tation masks, and reference edited images for training and evaluation. Our extensive experiments demonstrate the ef-fectiveness of our approach in identifying and editing tar-get objects, while conventional general image editing and region-based image editing methods have difficulties in this challenging task.
BibTeX
@inproceedings{liu2024referring,
  author = {Chang Liu and Xiangtai Li and Henghui Ding},
  title = {{Referring Image Editing: Object-Level Image Editing via Referring Expressions}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  url = {https://openreview.net/forum?id=dy8FN3nBkL},
}
Figure