Specify and Edit: Overcoming Ambiguity in Text-Based Image Editing

Ekaterina Iakovleva, Fabio Pizzati, Philip Torr, Stéphane LathuilièrearXiv 2024 · DOI · arXiv

Introduction

Text-conditioned diffusion models have revolutionized image editing, allowing users to modify images through simple natural language instructions. However, these models often struggle with ambiguity. When a user provides a vague prompt like "make the dog look cool" or "make the scene vintage," the model lacks the contextual reasoning to determine specific visual changes. This often leads to underwhelming results where the model either ignores the request or applies generic, low-quality edits. Furthermore, existing models operate as "black boxes," offering no transparency into the reasoning behind a specific edit.

The SANE Framework

To address these challenges, we introduce Specify ANd Edit (SANE), a zero-shot inference pipeline designed to resolve ambiguity without requiring additional training. SANE bridges the gap between abstract concepts and pixel-level interventions through a two-stage process.

1. Instruction Specification

When presented with an ambiguous prompt and an input image, SANE leverages a Large Language Model (LLM) to act as a "creative director." The LLM decomposes the vague instruction into a set of concrete, spatially-aware sub-tasks. For instance, "make the cat look funny" might be translated into specific instructions such as "add a hat," "add a bow tie," and "add a colored background."

2. Instruction Combination

The technical core of SANE is its novel spatial masking strategy for combining these sub-instructions. Rather than simple text-embedding averaging—which often dilutes detail—SANE allows instructions to "compete" for spatial dominance. By estimating the noise impact of each specific instruction at every denoising step, the model creates a mask that assigns each pixel to the most influential sub-instruction. This ensures that multiple edits can be applied simultaneously to different regions without interference.

The SANE pipeline: (Left) Instruction Specification using an LLM to decompose ambiguous prompts. (Right) The denoising loop where specific instructions are combined via spatial noise masking.

Experimental Results

SANE was evaluated on the EMU-Edit and IP2P datasets, focusing on samples with high ambiguity. The framework demonstrated significant improvements in both quantitative metrics and human-aligned qualitative assessments. On the EMU-Edit dataset, SANE improved the CLIP-based edit adherence metric by approximately 54% relative to the InstructPix2Pix baseline.

Furthermore, pairwise comparisons conducted via GPT-4o revealed a strong preference for SANE's outputs. Across various base models including MagicBrush and HQEdit, SANE's results were preferred 60-75% of the time, highlighting its ability to produce more diverse and relevant edits.

GPT-4o evaluation results showing the win rate of SANE (Ours) against baseline models across three major datasets.

Conclusion

SANE offers a powerful, training-free "wrapper" for existing image editing models. By combining the high-level reasoning of LLMs with a sophisticated spatial noise combination strategy, it transforms ambiguous user intent into precise visual modifications. This approach not only improves performance and diversity but also enhances the interpretability of the AI editing process.

Note: While SANE significantly improves quality, it does increase inference time linearly with the number of sub-instructions. Future work will focus on optimizing this computational overhead.

Summary updated

Abstract

Text-based editing diffusion models exhibit limited performance when the user's input instruction is ambiguous. To solve this problem, we propose $\textit{Specify ANd Edit}$ (SANE), a zero-shot inference pipeline for diffusion-based editing systems. We use a large language model (LLM) to decompose the input instruction into specific instructions, i.e. well-defined interventions to apply to the input image to satisfy the user's request. We benefit from the LLM-derived instructions along the original one, thanks to a novel denoising guidance strategy specifically designed for the task. Our experiments with three baselines and on two datasets demonstrate the benefits of SANE in all setups. Moreover, our pipeline improves the interpretability of editing models, and boosts the output diversity. We also demonstrate that our approach can be applied to any edit, whether ambiguous or not. Our code is public at https://github.com/fabvio/SANE.
BibTeX
@misc{iakovleva2024specify,
  author = {Ekaterina Iakovleva and Fabio Pizzati and Philip Torr and Stéphane Lathuilière},
  title = {{Specify and Edit: Overcoming Ambiguity in Text-Based Image Editing}},
  year = {2024},
  eprint = {2407.20232},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.20232},
}
Figure