TDRI: Two-Phase Dialogue Refinement and Co-Adaptation for Interactive Image Generation

Yuheng Feng, Jianhui Wang, Kun Li, Sida Li, Tianyu Shi, Haoyue Han, Miao Zhang, Xueqian Wang

TL;DR

TDRI (Two-Phase Dialogue Refinement and Co-Adaptation) is a novel interactive image generation framework that uses multi-turn dialogue to resolve ambiguous prompts and align outputs with user intent. By combining a dialogue-to-prompt module, a vision-language feedback loop, and an adaptive optimization strategy, TDRI significantly outperforms traditional single-turn prompt expansion methods, achieving a 33.6% human preference rate and a 45% reduction in iteration cycles.

What Problem Does This Paper Address?

Current text-to-image models (like Stable Diffusion or DALL-E 3) are incredibly powerful, but they frequently stumble when dealing with ambiguous prompts or complex user intents. Users often find themselves trapped in a frustrating trial-and-error loop, tweaking words blindly because the model fails to capture subtle textual nuances or precise layout requirements.

While using Large Language Models (LLMs) to augment or expand a prompt in a single turn helps, it doesn't solve the fundamental issue: the model cannot dynamically adapt to a user's unstated or evolving preferences. The authors of TDRI tackle this gap by turning image generation into a collaborative, multi-turn conversation between the human and the AI.

Methods

The TDRI framework splits the creative workflow into two distinct phases:

  1. Initial Generation Phase: The system processes the user's initial prompt to generate a base image. Crucially, it extracts a structural constraint—a set of pose keypoints—from this image using a pose estimator and applies Gaussian smoothing. This pose acts as a flexible anchor, maintaining core structural integrity across subsequent edits so the model doesn't completely redraw the scene from scratch.
  2. Interactive Refinement Phase: The image is iteratively updated across multiple dialogue turns using three core modules:
  3. Dialogue-to-Prompt (D2P): A summarizer model that synthesizes the entire dialogue history and the latest user feedback into a single, optimized prompt for the next generation round.
  4. Feedback-Reflection (FR): A Vision-Language Model (VLM, specifically Qwen-VL) inspects the generated image and creates a structured caption. The system then calculates an ambiguity score based on the mismatch between the VLM's description and the target prompt. If the score crosses a certain threshold, the system proactively asks the user for clarification (e.g., "Do you have a preferred color?").
  5. Adaptive Optimization (AO): To fine-tune the generation model (ControlNet) on the fly, the authors combine Direct Preference Optimization (DPO)—which maximizes the likelihood of generating user-preferred images over less-preferred ones—with Attend-and-Excite (A&E), which applies a loss function to ensure all key elements of the prompt are visually represented in the final image.
Figure 2: An overview of the two-phase framework TDRI. (a) In the Initial Generation Phase, the system processes user prompts via a U-Net-based diffusion model, generating base images with pose constraints. (b) In the Interactive Refinement Phase, user feedback is integrated to iteratively refine the image through dialogue-to-prompt generation, ambiguity scoring, and adaptive optimization.

Datasets

The framework was evaluated across two distinct scenarios using the following datasets:

  • DeepFashion Dataset: Used to fine-tune an SD-XL 1.0 base model for a fashion product creation task, allowing the model to master fine-grained clothing attributes, styles, and textures.
  • ImageReward Dataset: The authors curated 496 high-quality image-text pairs from this dataset, filtering out abstract or overly complex prompts, to benchmark general image generation.
  • User Prompts: Over 2,000 user-generated prompts were deployed to test multi-turn interactions, with each sample undergoing at least four dialogue rounds.

Key Insights and Ideas

  • Proactive Clarification Beats Guessing: Instead of blindly generating images from vague inputs, TDRI uses internal VLM feedback to calculate an explicit "ambiguity score." This allows the AI to recognize when it is confused and ask targeted questions, streamlining the alignment process.
  • The Sweet Spot of Interaction: Human evaluations revealed that user perception of the model's intent capture peaks around 3 rounds of dialogue, with diminishing returns after 5 to 6 rounds. This proves that tight alignment can be achieved quickly without exhausting the user.
  • Image Editing vs. From-Scratch: Iteratively editing an existing image (anchored by pose constraints) yields significantly higher user satisfaction (90%) and takes less time (9 minutes) than generating a brand-new image from scratch at each turn (78% satisfaction, 12 minutes).
  • Simplicity Wins: Simple prompts achieved a much higher generation success rate (92%) and CLIP score (0.85) than complex prompts (65% success, 0.60 CLIP score). This justifies TDRI's approach of breaking down complex, multi-layered user intents into simple, conversational steps.

Key Results

  • Human Preference: TDRI achieved a 33.6% human preference rate, vastly outperforming standard single-turn LLM prompt augmentation methods like GPT-4 augmentation (6.2%) and LLaMA-2 augmentation (6.1%).
  • Text-Image Alignment: TDRI scored the highest prompt-intent and image-intent alignment, reaching a T2I CLIPscore of 0.338 and an I2I CLIPscore of 0.812.
  • User Satisfaction: In iterative feedback tasks, user satisfaction reached 88% after 8 rounds of dialogue.
  • Efficiency: The framework demonstrated a 45% reduction in iteration cycles compared to conventional trial-and-error generation methods.

Limitations and Future Work

Despite its strong performance, TDRI has a few notable limitations:

  • Complex Prompt Handling: The model can still struggle to accurately translate highly complex, multi-level prompts into images because the underlying VLM sometimes fails to capture ultra-fine-grained details, leading to inaccurate internal feedback.
  • Cross-Modal Errors: Errors during the transfer between text and visual modalities can occasionally obscure user intent and lower communication efficiency.
  • Computational Cost: The iterative optimization loop (combining DPO and A&E on the fly) is computationally intensive and time-consuming, requiring high-end hardware (the experiments were conducted on Nvidia A6000 GPUs).

Future work will focus on enhancing the system's efficiency, expanding its ability to generalize across diverse inputs, and integrating finer feedback mechanisms.

Summary updated

Abstract

Although text-to-image generation technologies have made significant advancements, they still face challenges when dealing with ambiguous prompts and aligning outputs with user intent.Our proposed framework, TDRI (Two-Phase Dialogue Refinement and Co-Adaptation), addresses these issues by enhancing image generation through iterative user interaction. It consists of two phases: the Initial Generation Phase, which creates base images based on user prompts, and the Interactive Refinement Phase, which integrates user feedback through three key modules. The Dialogue-to-Prompt (D2P) module ensures that user feedback is effectively transformed into actionable prompts, which improves the alignment between user intent and model input. By evaluating generated outputs against user expectations, the Feedback-Reflection (FR) module identifies discrepancies and facilitates improvements. In an effort to ensure consistently high-quality results, the Adaptive Optimization (AO) module fine-tunes the generation process by balancing user preferences and maintaining prompt fidelity. Experimental results show that TDRI outperforms existing methods by achieving 33.6% human preference, compared to 6.2% for GPT-4 augmentation, and the highest CLIP and BLIP alignment scores (0.338 and 0.336, respectively). In iterative feedback tasks, user satisfaction increased to 88% after 8 rounds, with diminishing returns beyond 6 rounds. Furthermore, TDRI has been found to reduce the number of iterations and improve personalization in the creation of fashion products. TDRI exhibits a strong potential for a wide range of applications in the creative and industrial domains, as it streamlines the creative process and improves alignment with user preferences
BibTeX
@misc{feng2025tdri,
  author = {Yuheng Feng and Jianhui Wang and Kun Li and Sida Li and Tianyu Shi and Haoyue Han and Miao Zhang and Xueqian Wang},
  title = {{TDRI: Two-Phase Dialogue Refinement and Co-Adaptation for Interactive Image Generation}},
  year = {2025},
  eprint = {2503.17669},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.17669},
}
Figure