KnowVis: Dual-View Evaluation and Diagnosis of World-Knowledge Grounding in Text-to-Image Models

Yonghang Yu, Ruoyu Wang, Haoyu Zhang, Lin Qi, Yawei Zhou, Changqian YuarXiv 2026 · arXiv

TL;DR

KnowVis is a new dual-view benchmark designed to evaluate "world-knowledge grounding" in text-to-image (T2I) models—essentially checking if a model actually "knows" the facts, physics, and logic it's asked to draw. By combining a fine-grained "Skill-Tree" for atomic facts with a "Generalized" benchmark for complex scenarios, the authors reveal that even the most aesthetically pleasing models often fail basic factual and logical constraints.

What Problem Does This Paper Address?

Current text-to-image models are incredibly good at "drawing what is mentioned." If you ask for a "red sphere on a table," they rarely miss. However, real-world prompts often require more than surface-level alignment; they require world-knowledge grounding.

For example, if you prompt a model with "a convex lens converges parallel light rays," the challenge isn't just to draw a lens and some lines, but to draw them correctly according to the laws of optics. Existing evaluation metrics like CLIPScore or VQAScore often fail here because they focus on semantic similarity rather than factual correctness. There has been a lack of a unified framework that can diagnose whether a model's failure stems from a lack of knowledge, poor reasoning, or an inability to render complex annotations like formulas and diagrams.

Methods

The authors introduce a framework built on three pillars:

  1. Hierarchical Taxonomy: They developed an L1–L4 taxonomy (Domain $\rightarrow$ Subject $\rightarrow$ Branch $\rightarrow$ Knowledge Point). L4 represents "atomic" units of knowledge (e.g., a specific chemical reaction or a specific architectural style) that can be verified with a binary judgment.
  2. Dual-View Benchmarking:
    • Skill-Tree Benchmark: Focuses on L4 atomic points to diagnose specific knowledge gaps in a controlled setting.
  3. Generalized Benchmark: Targets open-ended, realistic scenarios where multiple pieces of knowledge must be composed (e.g., a science poster or a complex historical scene).
  4. Structured MLLM-as-a-Judge: Instead of asking a model "is this image correct?", they decompose each prompt into a checklist of Visual Question Answering (VQA) items. These are categorized as Required (essential for correctness), Optional (measuring richness), and Additional (dynamically discovered). To ensure the judge (Gemini 3.1 Pro Preview) is accurate, they provide it with "Knowledge Hints" and "Visual Grounding" (reference images from Wikimedia).
Figure 1: Overview of the KnowVis framework. It includes iterative taxonomy construction, dual- view benchmark construction, and structured VQA-based evaluation. The Skill-Tree Benchmark focuses on L4-level atomic knowledge diagnosis, while the Generalized Benchmark targets open- ended knowledge-grounded generation with human–judge collaborative evaluation.

Datasets

The paper introduces two primary datasets:

  • Skill-Tree Benchmark: 1,622 cases covering 240 L4 atomic knowledge points across Natural Science, Formal Science, and Human Civilization.
  • Generalized Benchmark: 550 cases covering 70 L3 branches, designed for more realistic and compositional evaluation.
  • General-Domain T2I: A 100-case control set used to distinguish general generation capability from specific knowledge grounding.

Key Insights and Ideas

  • Aesthetics $\neq$ Knowledge: One of the most striking findings is that models like Midjourney V7, which lead in aesthetic quality and human preference, often perform poorly on world-knowledge grounding. High-quality pixels do not imply high-quality facts.
  • The "Formal Science" Gap: Models struggle significantly more with Formal Sciences (Math, Logic, Systems) than with Natural Sciences. CLIPScore actually shows a negative correlation with human judgments in this domain, proving how misleading standard metrics can be for technical content.
  • Annotations are the Bottleneck: Models are generally good at rendering "Entities" (drawing a bird) but fail miserably at "Annotations" (labels, arrows, formulas). This suggests that the "language" of diagrams and technical illustrations is a major frontier for T2I research.
  • Knowledge Invocation vs. Rendering: By rewriting prompts to include explicit factual details, the authors found that pass rates improved significantly. This suggests that models often "know" the facts but fail to "invoke" them from a brief, implicit prompt.
Figure 4: Quantitative comparison with existing automatic metrics. (a) Correlation between au- tomatic metrics and human GT VQA pass rate across three L1 knowledge domains. (b) L2-level consistency pattern across 12 subjects, where KnowVis more closely follows the human GT profile than CLIPScore and VQAScore.

Key Results

  • Model Performance: Nano Banana 2 emerged as the top performer across both benchmarks, showing strong grounding in both atomic facts and complex compositions.
  • Metric Reliability: The KnowVis protocol achieved 94.51% VQA-level accuracy compared to human annotators, significantly outperforming general-purpose alignment metrics.
  • Diagnostic Power: The Skill-Tree heatmap (Figure 5 in the paper) reveals that while models might pass entity-related checks 80-90% of the time, their pass rate for "Annotation Relations" (e.g., an arrow pointing to the correct part of a diagram) can drop below 20%.

Limitations and Future Work

The authors acknowledge several areas for improvement:

  • Knowledge Richness: Quantifying "richness" (how much extra valid info a model provides) remains more subjective and harder to automate than basic correctness.
  • Perceptual Coupling: In human evaluations, it remains difficult for annotators to completely decouple their perception of image quality from their judgment of factual correctness.
  • Static Limitation: The current benchmark is limited to static images. Extending this to video generation or multi-turn interactive editing (where knowledge must be preserved across frames or turns) is a critical next step.
  • Judge Scaling: While Gemini 3.1 Pro is a strong judge, the protocol's reliability with smaller, more accessible open-source MLLMs is not yet established.

Summary updated

Abstract

Text-to-image (T2I) models have made substantial progress in visual realism, aesthetic quality, and instruction following. However, real-world prompts often go beyond explicit visual descriptions and require implicit facts, structured knowledge, and domain-specific commonsense. Existing evaluations mainly focus on explicit prompt-to-image semantic alignment or specific knowledge subdomains, lacking unified evaluation and diagnosis of world-knowledge grounding in T2I models. We introduce KnowVis for dual-view evaluation and diagnosis of world-knowledge grounding in T2I generation. Built on a hierarchical knowledge taxonomy with case-level annotations, KnowVis combines two complementary benchmark views with a structured MLLM-based VQA judge protocol. The Skill-Tree Benchmark supports fine-grained atomic knowledge evaluation, while the Generalized Benchmark examines open-ended knowledge composition and expression. The protocol uses Required VQAs to assess atomic knowledge correctness in the Skill-Tree Benchmark, and combines Required, Optional, and dynamically discovered knowledge expressions with human-judge collaborative evaluation in the Generalized Benchmark. We validate the protocol's reliability on the Skill-Tree validation split, enabling scalable evaluation and diagnostic error attribution. Experiments show that KnowVis reveals notable gaps in world-knowledge grounding across existing T2I models and provides structured diagnostic signals for knowledge-intensive image generation.
BibTeX
@misc{yu2026knowvis,
  author = {Yonghang Yu and Ruoyu Wang and Haoyu Zhang and Lin Qi and Yawei Zhou and Changqian Yu},
  title = {{KnowVis: Dual-View Evaluation and Diagnosis of World-Knowledge Grounding in Text-to-Image Models}},
  year = {2026},
  eprint = {2611.18101},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.18101},
}
Figure