When humans create art or edit photos, they rarely do it in a single, definitive action. Instead, the creative process is fundamentally iterative: an artist sketches a rough outline, evaluates the composition, adds details, and progressively refines the piece based on continuous feedback. Early generative architectures like the Deep Recurrent Attentive Writer (DRAW) proposed by Karol Gregor et al. in DRAW: A Recurrent Neural Network For Image Generation recognized this, using recurrent networks to construct images step-by-step.
However, modern text-to-image diffusion models have largely drifted toward a "one-shot" paradigm. A user provides a complete prompt, and the model generates a static image in a single open-loop pass. If the output is slightly misaligned, or if the user wants to make a sequential adjustment—such as adding an object or changing the lighting—they are forced to rewrite the entire prompt and restart the generation process. This lack of continuity has driven a major research shift toward iterative and conversational image editing, transforming visual generation from a rigid, single-turn transaction into a dynamic, multi-turn dialogue.
The Core Challenge: The Statelessness of One-Shot Models
Why can't we just feed a new instruction to a standard diffusion model? The fundamental issue is statelessness. Standard pipelines lack temporal memory or an awareness of the image's creation history. As noted by Daneul Kim et al. in Improving Editability in Image Generation with Layer-wise Memory, this leads to a form of "memory loss." If a user requests a subsequent modification, a standard model often regenerates the entire scene from scratch, completely altering the background, style, or identity of existing objects.
To achieve true multi-turn instruction following, a model must maintain visual coherence across dependent instructions, where each edit builds directly upon the state established in the previous turn. This requires tracking both the semantic intent of the conversation and the structural layout of the image across time.
Tracking State via Intermediate Representations
To prevent memory loss, researchers decouple high-level scene planning from low-level pixel generation. Instead of mapping text directly to pixels, models maintain an intermediate, symbolic representation of the scene state.
For instance, Gaurav Mittal et al. in Interactive Image Generation Using Scene Graphs utilized dynamic scene graphs, allowing users to add or manipulate objects incrementally while preserving global context. Similarly, Forrest Huang et al. in Scones: Towards Conversational Authoring of Sketches demonstrated that processing a sequence of previous scene states alongside new natural language instructions allows a system to move, resize, or delete elements reliably.
In modern diffusion architectures, this state tracking is often handled by Large Language Models (LLMs) acting as layout planners. In the training-free framework proposed by Long Lian et al. in LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models, an LLM generates a structured layout blueprint before the diffusion model paints it, providing a stable anchor for multi-round editing.
Targeted Editing and Long-Context Memory
When a user issues a conversational command like "make the jacket blue," the rest of the image should remain untouched. Monolithic regeneration fails here, so conversational editing systems employ targeted, editing-specific pipelines.
Shichao Ma et al. in DialogDraw: Image Generation and Editing System Based on Multi-Turn Dialogue show that visual coherence is best maintained by routing instructions to specialized drawing and editing pipelines that modify existing image data rather than regenerating it. To make this process seamless across long conversations, Zhen Han in ACE: All-round Creator and Editor Following Instructions via Diffusion Transformer treats image manipulation as a multi-turn dialogue with long-context memory, ensuring that subsequent edits strictly respect the visual state and modifications established in earlier turns.
Sequential Planning and Self-Correction Loops
Complex prompts often fail in one-shot models because the network must resolve spatial relationships, counting, and attribute binding simultaneously. Multi-turn frameworks solve this by turning generation into an internal or external conversation.
Instead of a single open-loop pass, Long Lian in Self-Correcting LLM-Controlled Diffusion Models introduces a closed-loop framework where an LLM inspects the generated output, verifies it against the prompt, and iteratively issues correction instructions. This mimics the human workflow of planning, execution, and critique, significantly reducing logical errors and prompt-following failures. For fine-grained tasks like facial retouching, Wen Xue et al. in RetouchGPT: LLM-based Interactive High-Fidelity Face Retouching via Imperfection Prompting show that translating user feedback into sequential "imperfection prompts" allows models to handle multi-stage adjustments far more effectively than one-shot translation models.
Evaluating the Multi-Turn Frontier
As the field transitions from single-turn commands to complex editing chains, traditional evaluation metrics fall short. Single-step benchmarks cannot capture how errors accumulate over multiple turns. To address this, Yiwei Ma et al. developed I2EBench: A Comprehensive Benchmark for Instruction-based Image Editing to standardize testing for multi-step sequences.
Furthermore, tracking consistency across an editing chain introduces new complexities in error detection, an open challenge highlighted by Ron Yosef et al. in EditInspector: A Benchmark for Evaluation of Text-Guided Image Edits. By curating high-quality sequential datasets like MagicBrush (Kai Zhang et al.), the community is building the scaffolding necessary to master true conversational visual intelligence.
Key Foundational Work
DRAW: A Recurrent Neural Network For Image Generation (2015) — contributed 1 concept elements to this direction.