Abstract
Language-Driven Image Editing (LDIE) allows users to perform complex image manipulations using natural language. This paper introduces the GIER dataset, a large-scale collection of 30,000 real-world editing requests and expert-annotated edits sourced from online communities. Alongside the dataset, the authors propose a modular baseline model that decomposes free-form instructions into a sequence of interpretable operations, such as object removal or brightness adjustment. Experimental results demonstrate that this modular approach is preferred by users over traditional black-box generative models.
Introduction
Professional image editing tools like Adobe Photoshop offer immense power but require significant technical expertise. Conversely, existing automated editing models are often limited to specific domains or global adjustments. There is a clear need for systems that can interpret free-form, multi-step, and grounded (local) requests on diverse, real-world images. This work addresses this gap by formalizing the LDIE task and providing the necessary data and modeling foundations.
The GIER Dataset
To facilitate research in LDIE, the authors curated the Grounded Image Editing Request (GIER) dataset. Unlike previous datasets that rely on synthetic changes or simple attribute swaps, GIER is built from real-world data crawled from "Photoshop request" communities on Reddit and Zhopped.com.
The dataset includes source/target image pairs, natural language requests from both amateurs and experts, and ground-truth masks for local operations. It stands out from older datasets because it features general user photos and complex, multi-operation requests, setting a new standard for realism in image editing benchmarks.
Methodology: A Modular Framework
Rather than employing a single "black-box" neural network, the authors propose a modular pipeline designed for interpretability and control. The framework consists of three primary stages:
- Operation Prediction: A multi-label classifier (ResNet18 for vision, Bi-LSTM for language) identifies which of 23 possible editing operations (e.g., contrast, inpainting, sharpening) are required.
- Operation Conditioned Grounding: The model determines if an operation should be applied globally or locally. For local edits, it generates a mask to ground the operation to a specific image region.
- Operation Modular Network (OMN): The execution layer constructs a chain of submodules. Each submodule—either a differentiable filter or a specialized network like EdgeConnect—predicts its own parameters based on the image and text context.
Key Results and Insights
The modular approach offers a significant advantage in interpretability. Users can inspect the specific operations performed and the parameters chosen, allowing for manual fine-tuning.
Human evaluation studies revealed a strong preference for the modular method over GAN-based baselines like Pix2pix. While Pix2pix might achieve lower pixel-wise error (L1), humans found the modular results more perceptually appealing. Notably, 86.5% of users preferred the modular output as a starting point for further interactive editing.
Ablation studies also highlighted that language requests are highly descriptive; the model could predict the correct operation almost as effectively using only the text request as it could using both text and image inputs.
Conclusion and Future Work
This work establishes a robust baseline for language-driven image editing. While the current model handles a wide array of operations, future research is needed to address the "long tail" of complex artistic filters and the addition of entirely new objects. Moving toward weakly supervised learning—where models learn from final target images without requiring intermediate masks—remains a promising direction for scaling these systems.
Summary updated