Introduction
Text-conditioned diffusion models have revolutionized image editing, allowing users to modify images through simple natural language instructions. However, these models often struggle with ambiguity. When a user provides a vague prompt like "make the dog look cool" or "make the scene vintage," the model lacks the contextual reasoning to determine specific visual changes. This often leads to underwhelming results where the model either ignores the request or applies generic, low-quality edits. Furthermore, existing models operate as "black boxes," offering no transparency into the reasoning behind a specific edit.
The SANE Framework
To address these challenges, we introduce Specify ANd Edit (SANE), a zero-shot inference pipeline designed to resolve ambiguity without requiring additional training. SANE bridges the gap between abstract concepts and pixel-level interventions through a two-stage process.
1. Instruction Specification
When presented with an ambiguous prompt and an input image, SANE leverages a Large Language Model (LLM) to act as a "creative director." The LLM decomposes the vague instruction into a set of concrete, spatially-aware sub-tasks. For instance, "make the cat look funny" might be translated into specific instructions such as "add a hat," "add a bow tie," and "add a colored background."
2. Instruction Combination
The technical core of SANE is its novel spatial masking strategy for combining these sub-instructions. Rather than simple text-embedding averaging—which often dilutes detail—SANE allows instructions to "compete" for spatial dominance. By estimating the noise impact of each specific instruction at every denoising step, the model creates a mask that assigns each pixel to the most influential sub-instruction. This ensures that multiple edits can be applied simultaneously to different regions without interference.
Experimental Results
SANE was evaluated on the EMU-Edit and IP2P datasets, focusing on samples with high ambiguity. The framework demonstrated significant improvements in both quantitative metrics and human-aligned qualitative assessments. On the EMU-Edit dataset, SANE improved the CLIP-based edit adherence metric by approximately 54% relative to the InstructPix2Pix baseline.
Furthermore, pairwise comparisons conducted via GPT-4o revealed a strong preference for SANE's outputs. Across various base models including MagicBrush and HQEdit, SANE's results were preferred 60-75% of the time, highlighting its ability to produce more diverse and relevant edits.
Conclusion
SANE offers a powerful, training-free "wrapper" for existing image editing models. By combining the high-level reasoning of LLMs with a sophisticated spatial noise combination strategy, it transforms ambiguous user intent into precise visual modifications. This approach not only improves performance and diversity but also enhances the interpretability of the AI editing process.
Note: While SANE significantly improves quality, it does increase inference time linearly with the number of sub-instructions. Future work will focus on optimizing this computational overhead.
Summary updated