Interactive Visual Assessment for Text-to-Image Generation Models

Xiaoyue Mi, Fan Tang, Juan Cao, Qiang Sheng, Ziyao Huang, Peng Li, Yang Liu, Tong-Yee LeearXiv 2024 · DOI · arXiv

Introduction

The rapid evolution of text-to-image (T2I) models, such as Stable Diffusion 3 (SD3), has outpaced the methodologies used to evaluate them. Traditional assessment relies on static datasets like MS-COCO, which present three significant limitations: fixed coverage that fails to capture the infinite variety of user prompts, potential data leakage where models are trained on the test sets themselves, and a lack of interpretability regarding why a specific generation failed. To address these gaps, we introduce DyEval, a dynamic, LLM-powered framework designed to interactively explore the capability boundaries of T2I models through a human-in-the-loop approach.

The DyEval Framework

DyEval treats model assessment as an iterative, hierarchical exploration rather than a one-off test. The framework is built around three core components: the Testing Tree, Dynamic Failure Location, and Contextual Reflection.

The Testing Tree and Exploration

Evaluation begins with a broad topic (e.g., "Spatial Relationships"). An LLM (GPT-3.5-turbo) generates specific test prompts. If the T2I model successfully generates images for these prompts—as verified by human annotators—the LLM progresses deeper into the tree, generating more complex and fine-grained sub-topics. This ensures that the evaluation continuously pushes the model toward its breaking points.

The DyEval workflow: from initial test context to dynamic failure location and contextual reflection.

Dynamic Failure Location and Reflection

When a model fails, DyEval employs a "divide-and-conquer" strategy to isolate the cause. It converts the failed prompt into a structured Scene Graph and systematically simplifies it to identify the "minimal failure subset." For instance, if a complex prompt involving multiple objects and relations fails, this module can determine if the failure was triggered by a specific relationship (e.g., "looking at") or a particular object attribute. Finally, the LLM analyzes the history of passes and failures to generate a "reflection" report, identifying high-level patterns in the model's weaknesses.

Experimental Evaluation

We utilized DyEval to generate a massive evaluation corpus of 24,054 text-image pairs across nine dimensions, including Material Objects, Abstract Objects, Explicit/Implicit Relations, and Cultural contexts. We tested four major models: Stable Diffusion v1.5, v2.1, XL (SDXL), and v3 (SD3). To ensure ground truth, 23 participants provided manual annotations for the generated images.

Comparative Performance

Our results demonstrate that DyEval is significantly more effective at identifying model weaknesses than non-adaptive methods. By actively searching for failure cases, DyEval identified up to 2.56 times more generation failures than static benchmarks using the same number of test cases. This is reflected in the significantly lower pass rates observed under the DyEval framework compared to standard testing.

Comparison of pass rates between non-adaptive testing and DyEval across different Stable Diffusion models.

Insights into Model Failures

Beyond simple pass/fail metrics, DyEval provides deep insights into the linguistic and conceptual triggers of T2I failures:

  • The Superior Model Paradox: While newer models like SD3 and SDXL are generally more robust, DyEval successfully uncovered rare failure patterns, particularly in nuanced cultural contexts (e.g., traditional clothing) that static tests would likely overlook.
  • Linguistic Weak Points: Part-of-Speech (PoS) analysis revealed that Possessive Pronouns (PRP$) like "its" or "their" are frequent failure triggers. These words often lead to "pronoun-induced degradation," where the model fails to correctly attribute specific traits to the intended objects.
  • Abstract vs. Material: Models surprisingly performed better on "Abstract Objects" (emotions, ideas) than "Material Objects." This suggests that abstract concepts allow for more creative leeway, whereas material objects require strict physical and anatomical correctness that is easier for human evaluators to flag as failures.

Conclusion and Future Work

DyEval represents a shift from static benchmarking to dynamic debugging for generative AI. By isolating specific failure triggers, it provides researchers with actionable data to improve model logic and robustness. While the current framework requires significant human effort for assessment, future work will focus on integrating Vision-Language Models (VLMs) to automate the evaluation loop. Additionally, we aim to expand this framework to text-to-video models and develop automated "bug repair" mechanisms to fine-tune models based on DyEval's diagnostic reflections.

Summary updated

Abstract

Visual generation models have achieved remarkable progress in computer graphics applications but still face significant challenges in real-world deployment. Current assessment approaches for visual generation tasks typically follow an isolated three-phase framework: test input collection, model output generation, and user assessment. These fashions suffer from fixed coverage, evolving difficulty, and data leakage risks, limiting their effectiveness in comprehensively evaluating increasingly complex generation models. To address these limitations, we propose DyEval, an LLM-powered dynamic interactive visual assessment framework that facilitates collaborative evaluation between humans and generative models for text-to-image systems. DyEval features an intuitive visual interface that enables users to interactively explore and analyze model behaviors, while adaptively generating hierarchical, fine-grained, and diverse textual inputs to continuously probe the capability boundaries of the models based on their feedback. Additionally, to provide interpretable analysis for users to further improve tested models, we develop a contextual reflection module that mines failure triggers of test inputs and reflects model potential failure patterns supporting in-depth analysis using the logical reasoning ability of LLM. Qualitative and quantitative experiments demonstrate that DyEval can effectively help users identify max up to 2.56 times generation failures than conventional methods, and uncover complex and rare failure patterns, such as issues with pronoun generation and specific cultural context generation. Our framework provides valuable insights for improving generative models and has broad implications for advancing the reliability and capabilities of visual generation systems across various domains.
BibTeX
@misc{mi2024interactive,
  author = {Xiaoyue Mi and Fan Tang and Juan Cao and Qiang Sheng and Ziyao Huang and Peng Li and Yang Liu and Tong-Yee Lee},
  title = {{Interactive Visual Assessment for Text-to-Image Generation Models}},
  year = {2024},
  eprint = {2411.15509},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.15509},
}
Figure