Introduction
The rapid evolution of text-to-image (T2I) models, such as Stable Diffusion 3 (SD3), has outpaced the methodologies used to evaluate them. Traditional assessment relies on static datasets like MS-COCO, which present three significant limitations: fixed coverage that fails to capture the infinite variety of user prompts, potential data leakage where models are trained on the test sets themselves, and a lack of interpretability regarding why a specific generation failed. To address these gaps, we introduce DyEval, a dynamic, LLM-powered framework designed to interactively explore the capability boundaries of T2I models through a human-in-the-loop approach.
The DyEval Framework
DyEval treats model assessment as an iterative, hierarchical exploration rather than a one-off test. The framework is built around three core components: the Testing Tree, Dynamic Failure Location, and Contextual Reflection.
The Testing Tree and Exploration
Evaluation begins with a broad topic (e.g., "Spatial Relationships"). An LLM (GPT-3.5-turbo) generates specific test prompts. If the T2I model successfully generates images for these prompts—as verified by human annotators—the LLM progresses deeper into the tree, generating more complex and fine-grained sub-topics. This ensures that the evaluation continuously pushes the model toward its breaking points.
Dynamic Failure Location and Reflection
When a model fails, DyEval employs a "divide-and-conquer" strategy to isolate the cause. It converts the failed prompt into a structured Scene Graph and systematically simplifies it to identify the "minimal failure subset." For instance, if a complex prompt involving multiple objects and relations fails, this module can determine if the failure was triggered by a specific relationship (e.g., "looking at") or a particular object attribute. Finally, the LLM analyzes the history of passes and failures to generate a "reflection" report, identifying high-level patterns in the model's weaknesses.
Experimental Evaluation
We utilized DyEval to generate a massive evaluation corpus of 24,054 text-image pairs across nine dimensions, including Material Objects, Abstract Objects, Explicit/Implicit Relations, and Cultural contexts. We tested four major models: Stable Diffusion v1.5, v2.1, XL (SDXL), and v3 (SD3). To ensure ground truth, 23 participants provided manual annotations for the generated images.
Comparative Performance
Our results demonstrate that DyEval is significantly more effective at identifying model weaknesses than non-adaptive methods. By actively searching for failure cases, DyEval identified up to 2.56 times more generation failures than static benchmarks using the same number of test cases. This is reflected in the significantly lower pass rates observed under the DyEval framework compared to standard testing.
Insights into Model Failures
Beyond simple pass/fail metrics, DyEval provides deep insights into the linguistic and conceptual triggers of T2I failures:
- The Superior Model Paradox: While newer models like SD3 and SDXL are generally more robust, DyEval successfully uncovered rare failure patterns, particularly in nuanced cultural contexts (e.g., traditional clothing) that static tests would likely overlook.
- Linguistic Weak Points: Part-of-Speech (PoS) analysis revealed that Possessive Pronouns (PRP$) like "its" or "their" are frequent failure triggers. These words often lead to "pronoun-induced degradation," where the model fails to correctly attribute specific traits to the intended objects.
- Abstract vs. Material: Models surprisingly performed better on "Abstract Objects" (emotions, ideas) than "Material Objects." This suggests that abstract concepts allow for more creative leeway, whereas material objects require strict physical and anatomical correctness that is easier for human evaluators to flag as failures.
Conclusion and Future Work
DyEval represents a shift from static benchmarking to dynamic debugging for generative AI. By isolating specific failure triggers, it provides researchers with actionable data to improve model logic and robustness. While the current framework requires significant human effort for assessment, future work will focus on integrating Vision-Language Models (VLMs) to automate the evaluation loop. Additionally, we aim to expand this framework to text-to-video models and develop automated "bug repair" mechanisms to fine-tune models based on DyEval's diagnostic reflections.
Summary updated