TL;DR
TDRI (Two-Phase Dialogue Refinement and Co-Adaptation) is a novel interactive image generation framework that uses multi-turn dialogue to resolve ambiguous prompts and align outputs with user intent. By combining a dialogue-to-prompt module, a vision-language feedback loop, and an adaptive optimization strategy, TDRI significantly outperforms traditional single-turn prompt expansion methods, achieving a 33.6% human preference rate and a 45% reduction in iteration cycles.
What Problem Does This Paper Address?
Current text-to-image models (like Stable Diffusion or DALL-E 3) are incredibly powerful, but they frequently stumble when dealing with ambiguous prompts or complex user intents. Users often find themselves trapped in a frustrating trial-and-error loop, tweaking words blindly because the model fails to capture subtle textual nuances or precise layout requirements.
While using Large Language Models (LLMs) to augment or expand a prompt in a single turn helps, it doesn't solve the fundamental issue: the model cannot dynamically adapt to a user's unstated or evolving preferences. The authors of TDRI tackle this gap by turning image generation into a collaborative, multi-turn conversation between the human and the AI.
Methods
The TDRI framework splits the creative workflow into two distinct phases:
- Initial Generation Phase: The system processes the user's initial prompt to generate a base image. Crucially, it extracts a structural constraint—a set of pose keypoints—from this image using a pose estimator and applies Gaussian smoothing. This pose acts as a flexible anchor, maintaining core structural integrity across subsequent edits so the model doesn't completely redraw the scene from scratch.
- Interactive Refinement Phase: The image is iteratively updated across multiple dialogue turns using three core modules:
- Dialogue-to-Prompt (D2P): A summarizer model that synthesizes the entire dialogue history and the latest user feedback into a single, optimized prompt for the next generation round.
- Feedback-Reflection (FR): A Vision-Language Model (VLM, specifically Qwen-VL) inspects the generated image and creates a structured caption. The system then calculates an ambiguity score based on the mismatch between the VLM's description and the target prompt. If the score crosses a certain threshold, the system proactively asks the user for clarification (e.g., "Do you have a preferred color?").
- Adaptive Optimization (AO): To fine-tune the generation model (ControlNet) on the fly, the authors combine Direct Preference Optimization (DPO)—which maximizes the likelihood of generating user-preferred images over less-preferred ones—with Attend-and-Excite (A&E), which applies a loss function to ensure all key elements of the prompt are visually represented in the final image.
Datasets
The framework was evaluated across two distinct scenarios using the following datasets:
- DeepFashion Dataset: Used to fine-tune an SD-XL 1.0 base model for a fashion product creation task, allowing the model to master fine-grained clothing attributes, styles, and textures.
- ImageReward Dataset: The authors curated 496 high-quality image-text pairs from this dataset, filtering out abstract or overly complex prompts, to benchmark general image generation.
- User Prompts: Over 2,000 user-generated prompts were deployed to test multi-turn interactions, with each sample undergoing at least four dialogue rounds.
Key Insights and Ideas
- Proactive Clarification Beats Guessing: Instead of blindly generating images from vague inputs, TDRI uses internal VLM feedback to calculate an explicit "ambiguity score." This allows the AI to recognize when it is confused and ask targeted questions, streamlining the alignment process.
- The Sweet Spot of Interaction: Human evaluations revealed that user perception of the model's intent capture peaks around 3 rounds of dialogue, with diminishing returns after 5 to 6 rounds. This proves that tight alignment can be achieved quickly without exhausting the user.
- Image Editing vs. From-Scratch: Iteratively editing an existing image (anchored by pose constraints) yields significantly higher user satisfaction (90%) and takes less time (9 minutes) than generating a brand-new image from scratch at each turn (78% satisfaction, 12 minutes).
- Simplicity Wins: Simple prompts achieved a much higher generation success rate (92%) and CLIP score (0.85) than complex prompts (65% success, 0.60 CLIP score). This justifies TDRI's approach of breaking down complex, multi-layered user intents into simple, conversational steps.
Key Results
- Human Preference: TDRI achieved a 33.6% human preference rate, vastly outperforming standard single-turn LLM prompt augmentation methods like GPT-4 augmentation (6.2%) and LLaMA-2 augmentation (6.1%).
- Text-Image Alignment: TDRI scored the highest prompt-intent and image-intent alignment, reaching a T2I CLIPscore of 0.338 and an I2I CLIPscore of 0.812.
- User Satisfaction: In iterative feedback tasks, user satisfaction reached 88% after 8 rounds of dialogue.
- Efficiency: The framework demonstrated a 45% reduction in iteration cycles compared to conventional trial-and-error generation methods.
Limitations and Future Work
Despite its strong performance, TDRI has a few notable limitations:
- Complex Prompt Handling: The model can still struggle to accurately translate highly complex, multi-level prompts into images because the underlying VLM sometimes fails to capture ultra-fine-grained details, leading to inaccurate internal feedback.
- Cross-Modal Errors: Errors during the transfer between text and visual modalities can occasionally obscure user intent and lower communication efficiency.
- Computational Cost: The iterative optimization loop (combining DPO and A&E on the fly) is computationally intensive and time-consuming, requiring high-end hardware (the experiments were conducted on Nvidia A6000 GPUs).
Future work will focus on enhancing the system's efficiency, expanding its ability to generalize across diverse inputs, and integrating finer feedback mechanisms.
Summary updated