Abstract
Despite the impressive success of text-to-image (TTI) models, existing studies overlook the issue of whether these models accurately convey factual information. In this paper, we focus on the problem of image hallucination, where images created by TTI models fail to faithfully depict factual content. To address this, we introduce I-HallA (Image Hallucination evaluation with Question Answering), a novel automated evaluation metric that measures the factuality of generated images through visual question answering (VQA). We also introduce I-HallA v1.0, a curated benchmark dataset for this purpose. As part of this process, we develop a pipeline that generates high-quality question-answer pairs using multiple GPT-4 Omni-based agents, with human judgments to ensure accuracy. Our evaluation protocols measure image hallucination by testing if images from existing TTI models can correctly respond to these questions. The I-HallA v1.0 dataset comprises 1.2K diverse image-text pairs across nine categories with 1,000 rigorously curated questions covering various compositional challenges. We evaluate five TTI models using I-HallA and reveal that these state-of-the-art models often fail to accurately convey factual information. Moreover, we validate the reliability of our metric by demonstrating a strong Spearman correlation (ρ=0.95) with human judgments. Our benchmark dataset and metric can serve as a foundation for developing factually accurate TTI models.
Introduction
The rapid advancement of text-to-image (TTI) generation has enabled the creation of visually stunning images from simple text prompts. However, current evaluation metrics primarily focus on prompt alignment—whether the image matches the description—rather than factual accuracy. This gap leads to "image hallucinations," where a model generates a visually plausible image that contradicts established facts. For example, a model might correctly generate a "molecule" but fail to include the correct number of atoms required by chemistry.
To bridge this gap, we propose I-HallA, a framework that uses Visual Question Answering (VQA) to probe the factual integrity of generated images. By asking specific questions that require external knowledge (e.g., "How many hydrogen atoms are connected to the central atom?"), I-HallA can distinguish between a factually correct image and a hallucinated one.
Methods: The I-HallA Framework
The I-HallA framework utilizes a multi-stage pipeline to evaluate TTI models. The process begins with the curation of prompts from authoritative sources like textbooks, ensuring a high standard of factual ground truth.
The pipeline involves two main components:
- Reasoning Enhancement: Using GPT-4o, we generate detailed reasoning for why an image is factual or hallucinated. This step incorporates external knowledge that may not be explicitly present in the initial prompt.
- Automated VQA Evaluation: We generate 1,000 multiple-choice question-answer (QA) sets. To evaluate a TTI model, its generated images are processed by a VQA agent (e.g., GPT-4o) tasked with answering these questions. The final I-HallA score is the ratio of correct answers, providing a quantitative measure of factual accuracy.
The I-HallA v1.0 Benchmark
We introduce the I-HallA v1.0 benchmark, which consists of 1,200 image-text pairs derived from 200 prompts across nine categories in Science and History. The dataset includes:
- Science: Physics, Biology, and Earth Science.
- History: Ancient, Medieval, and Modern eras across various global regions.
- Compositional Challenges: Questions are categorized into "Compositions of Interest" (CoIs) such as counting, existence, posture, and spatial relations.
Key Results
Our evaluation of five state-of-the-art TTI models (Stable Diffusion variants and DALL-E 3) reveals significant shortcomings in factual accuracy:
- DALL-E 3 Performance: While DALL-E 3 leads the field, it only achieves an I-HallA score of approximately 0.66, indicating frequent factual errors.
- Strict Evaluation: Under a stricter scoring regime (requiring all questions for a prompt to be correct), DALL-E 3’s performance drops significantly (0.22 in Science, 0.13 in History).
- Human Correlation: The I-HallA metric demonstrates a high Spearman correlation (ρ=0.95) with human expert judgments, validating its effectiveness as an automated proxy.
Conclusion
I-HallA provides a rigorous, automated method for evaluating the factuality of TTI models. Our findings highlight that visual plausibility does not equate to factual truth. By providing a benchmark grounded in textbook knowledge, we aim to encourage the development of TTI models that are not only creative but also factually reliable for educational and professional applications.
Summary updated