A Benchmark and Baseline for Language-Driven Image Editing

Jing Shi, Ning Xu, Trung Bui, Franck Dernoncourt, Zheng Wen, Chenliang Xu

Abstract

Language-Driven Image Editing (LDIE) allows users to perform complex image manipulations using natural language. This paper introduces the GIER dataset, a large-scale collection of 30,000 real-world editing requests and expert-annotated edits sourced from online communities. Alongside the dataset, the authors propose a modular baseline model that decomposes free-form instructions into a sequence of interpretable operations, such as object removal or brightness adjustment. Experimental results demonstrate that this modular approach is preferred by users over traditional black-box generative models.

Introduction

Professional image editing tools like Adobe Photoshop offer immense power but require significant technical expertise. Conversely, existing automated editing models are often limited to specific domains or global adjustments. There is a clear need for systems that can interpret free-form, multi-step, and grounded (local) requests on diverse, real-world images. This work addresses this gap by formalizing the LDIE task and providing the necessary data and modeling foundations.

The GIER Dataset

To facilitate research in LDIE, the authors curated the Grounded Image Editing Request (GIER) dataset. Unlike previous datasets that rely on synthetic changes or simple attribute swaps, GIER is built from real-world data crawled from "Photoshop request" communities on Reddit and Zhopped.com.

Overview of the GIER dataset, showing the source image, user request, target image, and expert annotations for operations and grounding masks.

The dataset includes source/target image pairs, natural language requests from both amateurs and experts, and ground-truth masks for local operations. It stands out from older datasets because it features general user photos and complex, multi-operation requests, setting a new standard for realism in image editing benchmarks.

Methodology: A Modular Framework

Rather than employing a single "black-box" neural network, the authors propose a modular pipeline designed for interpretability and control. The framework consists of three primary stages:

  1. Operation Prediction: A multi-label classifier (ResNet18 for vision, Bi-LSTM for language) identifies which of 23 possible editing operations (e.g., contrast, inpainting, sharpening) are required.
  2. Operation Conditioned Grounding: The model determines if an operation should be applied globally or locally. For local edits, it generates a mask to ground the operation to a specific image region.
  3. Operation Modular Network (OMN): The execution layer constructs a chain of submodules. Each submodule—either a differentiable filter or a specialized network like EdgeConnect—predicts its own parameters based on the image and text context.
The proposed modular framework, illustrating the flow from input request to operation prediction, grounding, and final execution via the Operation Modular Network.

Key Results and Insights

The modular approach offers a significant advantage in interpretability. Users can inspect the specific operations performed and the parameters chosen, allowing for manual fine-tuning.

Human evaluation studies revealed a strong preference for the modular method over GAN-based baselines like Pix2pix. While Pix2pix might achieve lower pixel-wise error (L1), humans found the modular results more perceptually appealing. Notably, 86.5% of users preferred the modular output as a starting point for further interactive editing.

Ablation studies also highlighted that language requests are highly descriptive; the model could predict the correct operation almost as effectively using only the text request as it could using both text and image inputs.

Conclusion and Future Work

This work establishes a robust baseline for language-driven image editing. While the current model handles a wide array of operations, future research is needed to address the "long tail" of complex artistic filters and the addition of entirely new objects. Moving toward weakly supervised learning—where models learn from final target images without requiring intermediate masks—remains a promising direction for scaling these systems.

Summary updated

BibTeX
@inproceedings{shi2020benchmark,
  author = {Jing Shi and Ning Xu and Trung Bui and Franck Dernoncourt and Zheng Wen and Chenliang Xu},
  title = {{A Benchmark and Baseline for Language-Driven Image Editing}},
  booktitle = {ACCV},
  year = {2020},
  url = {https://openreview.net/forum?id=o8iOyWiZoNe},
}
Figure