Abstract
Despite the impressive success of text-to-image (TTI) generation models, existing studies overlook the issue of whether these models accurately convey factual information. In this paper, we focus on the problem of image hallucination, where images created by generation models fail to faithfully depict factual content. To address this, we introduce I-HallA (Image Hallucination evaluation with Question Answering), a novel automated evaluation metric that measures the factuality of generated images through visual question answering (VQA). We also introduce I-HallA v1.0, a curated benchmark dataset for this purpose. As part of this process, we develop a pipeline that generates high-quality question-answer pairs using multiple GPT-4 Omni-based agents, with human judgments to ensure accuracy. Our evaluation protocols measure image hallucination by testing if images from existing text-to-image models can correctly respond to these questions. The I-HallA v1.0 dataset comprises 1.2K diverse image-text pairs across nine categories with 1,000 rigorously curated questions covering various compositional challenges. We evaluate five text-to-image models using I-HallA and reveal that these state-of-the-art models often fail to accurately convey factual information. Moreover, we validate the reliability of our metric by demonstrating a strong Spearman correlation (rho=0.95) with human judgments. We believe our benchmark dataset and metric can serve as a foundation for developing factually accurate text-to-image generation models.
Introduction
While current text-to-image (TTI) models are incredibly capable, they often suffer from a specific failure mode termed image hallucination. Unlike simple misalignment—where a model might ignore a specific keyword in a prompt—image hallucination occurs when a model generates a visually plausible image that is factually incorrect.
For example, if a user requests "a molecule of methane," a model might generate a high-quality 3D render. However, if that molecule depicts only three hydrogen atoms instead of the required four, it is a hallucination. Existing metrics like TIFA or CLIPScore focus on whether the image matches the text of the prompt but lack the external knowledge (e.g., chemistry, history) to verify if the depicted content is actually true. This gap is critical for AI applications in education, journalism, and technical fields.
The I-HallA Framework
To address this, we propose I-HallA (Image Hallucination evaluation with Question Answering), a framework that leverages Vision-Language Models (VLMs) to test the factuality of generated images through multiple-choice questions. The framework is built upon a three-stage pipeline:
- Dataset Collection: We curated 200 prompts and "factual" ground-truth images from authoritative science and history textbooks. We then generated "hallucinated" counterparts using five popular TTI models.
- Reasoning Enhancement: Using GPT-4o, we generated "factual reasoning" for each prompt—detailed explanations of the external knowledge required to depict the subject accurately (e.g., the specific number of atoms in a molecule or the correct color of the Statue of Liberty in 1890).
- QA Generation & Evaluation: We utilized a multi-agent GPT-4o setup to create five multiple-choice questions per image. These questions target specific "Compositions of Interest" (CoIs) such as counting, spatial relations, and shapes.
To calculate the I-HallA score, a VLM acts as the "test-taker," answering these questions based on the generated image. The final score (ranging from 0 to 1) represents the percentage of correct factual answers provided by the model.
Benchmark Dataset: I-HallA v1.0
The introduced benchmark consists of 1.2K image-text pairs and 1,000 rigorously curated QA sets. These cover nine diverse categories, including Physics, Biology, Earth Science, and various historical periods. Questions are categorized by the specific compositional challenge they test, such as Counting, Existence, Posture, Relation, and Visual Semantics.
Key Results and Insights
Our evaluation of five state-of-the-art models (Stable Diffusion v1.4, v1.5, v2.0, XL, and DALL-E 3) revealed several key findings:
- Prompt Alignment $\neq$ Factual Accuracy: A model can follow a prompt perfectly (e.g., "a comb-pattern pottery") but still hallucinate the specific historical pattern, rendering the image useless for educational purposes. This is often due to the "Polysemy Trap," where ambiguous text prompts lead models to generate visually similar but factually incorrect motifs.
- Model Performance: DALL-E 3 emerged as the top performer but still struggled with factual accuracy, scoring approximately 0.66 on the I-HallA scale. In contrast, factual images from textbooks scored ~0.86 (not 1.0 due to VLM reasoning noise). Stable Diffusion models struggled significantly, often scoring between 0.30 and 0.58.
- Metric Reliability: The I-HallA score demonstrates a very high Spearman correlation ($\rho=0.95$) with human judgment, confirming its effectiveness as an automated proxy for human evaluation.
Conclusion
I-HallA provides a robust foundation for evaluating the factuality of TTI models. While current models excel at visual aesthetics, they frequently fail to convey accurate information. Future work will involve expanding the benchmark to other fact-heavy domains like medicine and geography, and exploring how these evaluation insights can be used to fine-tune models to reduce hallucinations during the generation process.
Summary updated