Abstract
We introduce "Idea to Image," a system that enables multimodal iterative self-refinement with GPT-4V(ision) for automatic image design and generation. Humans can quickly identify the characteristics of different text-to-image (T2I) models via iterative explorations. This enables them to efficiently convert their high-level generation ideas into effective T2I prompts that can produce good images. We investigate if systems based on large multimodal models (LMMs) can develop analogous multimodal self-refinement abilities that enable exploring unknown models or environments via self-refining tries. Idea2Img cyclically generates revised T2I prompts to synthesize draft images, and provides directional feedback for prompt revision, both conditioned on its memory of the probed T2I model's characteristics. The iterative self-refinement brings Idea2Img various advantages over vanilla T2I models. Notably, Idea2Img can process input ideas with interleaved image-text sequences, follow ideas with design instructions, and generate images of better semantic and visual qualities. The user preference study validates the efficacy of multimodal iterative self-refinement on automatic image design and generation.
Introduction
While text-to-image (T2I) models like Stable Diffusion and DALL-E have reached impressive levels of fidelity, they remain notoriously difficult to control. Users often find themselves trapped in a cycle of "prompt engineering"—a tedious process of trial and error required to make models respect specific object counts, maintain stylistic consistency from reference images, or follow complex spatial instructions.
Most T2I models are essentially "blind" to the user's original intent once a prompt is submitted. If the output fails to capture a nuance, the user must manually diagnose the failure and rewrite the prompt. Idea2Img addresses this by automating the human-in-the-loop exploration process. It allows users to provide "multimodal IDEAs" (interleaved text and images) and high-level design instructions, acting as an agentic prompt engineer that refines its own output.
The Idea2Img Framework
The core of Idea2Img is a multimodal iterative loop powered by GPT-4V(ision). Rather than a single-shot generation, the framework treats the T2I model as an environment to be explored and mastered through four key stages:
- Prompt Generation: GPT-4V analyzes the user's multimodal IDEA and generates $N$ distinct, detailed T2I prompts.
- Draft Image Selection: After the T2I model generates draft images, GPT-4V evaluates them and selects the most promising candidate that aligns with the original IDEA, effectively acting as a noise-reduction filter.
- Feedback Reflection: The agent compares the selected draft to the original IDEA, identifying specific discrepancies (e.g., incorrect breeds, misspelled text, or missing objects) to provide directional feedback.
- Memory Module: The system maintains a history of previous iterations, allowing the LMM to "learn" the specific quirks of the underlying T2I model. For example, if a model fails to understand a technical term, the memory module helps the agent pivot to more literal physical descriptions in the next round.
Experimental Evaluation
To evaluate the system's performance on complex design tasks, the authors introduced the Idea2Img Evaluation Set, a benchmark consisting of 104 challenging multimodal IDEA samples. This set includes 33 text-only queries, 43 queries with a single reference image, and 28 queries with multiple reference images. The benchmark covers difficult scenarios such as object counting, scene text, style transfer, concept customization, and negative constraints (e.g., "a plate with no bananas").
Key Results and Insights
Idea2Img demonstrates significant improvements across a wide array of visual tasks. In a head-to-head user preference study, images generated by Idea2Img were overwhelmingly preferred over those from manual human-written prompts. When paired with SDXL v1.0, the framework improved user preference scores by +26.9% compared to initial-round prompts.
Several key insights emerged from the study:
- LMMs as Tool Explorers: GPT-4V acts as an explorer rather than a simple translator, probing the T2I model to discover which linguistic structures yield the best results for that specific engine.
- The Power of Selection: The ability to pick the best of $N$ drafts is crucial. Because T2I models are stochastic, "bad luck" generations often derail simple prompt-expansion techniques; selection mitigates this variance.
- Scaling with Model Capability: Interestingly, more capable T2I models (like SDXL) saw a larger performance boost from Idea2Img than weaker models. This suggests that stronger models possess latent potential that requires sophisticated prompting to unlock.
Limitations and Future Work
Despite its efficacy, Idea2Img faces challenges regarding latency and cost. Running multiple iterations of GPT-4V and generating multiple T2I drafts per cycle is significantly more resource-intensive than single-shot generation. Additionally, there remains a performance gap between proprietary models like GPT-4V and current open-source LMMs, which struggle with complex reasoning and long-term memory in iterative loops.
Future research directions include extending this self-refining agent architecture to other domains, such as video generation, GUI navigation, and embodied AI agents that must learn to interact with unknown environments through trial and error.
Summary updated