TL;DR
While text-to-image (TTI) models can generate breathtaking artwork, they frequently fail at a critical task: accurately depicting factual reality. This paper introduces I-HallA, a novel evaluation framework and benchmark that uses Visual Question Answering (VQA) to detect "image hallucinations"—instances where generated images contradict external factual knowledge. By evaluating models on textbook-derived facts, I-HallA provides an automated metric that correlates remarkably well ($\rho = 0.95$) with human judgment.
What Problem Does This Paper Address?
State-of-the-art text-to-image models (like DALL-E 3 and Stable Diffusion) are incredibly good at text alignment—making sure that if you ask for a "molecule," you get a molecule. However, existing evaluation metrics stop there. They fail to check if the generated image is factually accurate based on real-world knowledge.
The authors define this gap as image hallucination. For example, if a user prompts a model to generate "a molecule of methane," standard metrics check if a molecule is present. But they won't notice if the model generates five hydrogen atoms instead of the factually correct four.
This lack of factual integrity poses serious risks of spreading misinformation, particularly when these generative models are deployed to create educational materials or media content.
Methods
To systematically evaluate image hallucination, the authors developed a three-stage pipeline to build the I-HallA v1.0 benchmark and evaluation metric:
- Dataset Collection: The team curated 200 highly factual prompts directly from authoritative science and history textbooks, pairing each with its true "factual image" and hallucinated images generated by five different TTI models.
- Dataset Enhancement via LLM/VLM Reasoning: They fed the prompts and images into GPT-4o to extract detailed "reasoning" behind why an image is factual or hallucinated. This step injects external domain knowledge and visual semantics that aren't explicitly stated in the prompt text. Ten human annotators rigorously reviewed and refined these reasonings.
- QA Generation & VQA Evaluation: Using the refined reasoning, an LLM agent generated five multiple-choice questions per prompt, categorized by "Compositions of Interest" (CoIs) such as counting, shape, relation, and size.
To score a TTI model, its generated image is passed to a VQA model (GPT-4o) along with these five questions. The final I-HallA score is simply the ratio of correct answers, ranging from 0 (completely hallucinated) to 1 (factually flawless).
Datasets
The paper introduces the I-HallA v1.0 benchmark dataset, which consists of:
- 1.2K diverse image-text pairs split across nine textbook categories (including Physics, Biology, Earth Science, and various historical eras).
- 1,000 rigorously curated multiple-choice questions designed to probe specific factual and compositional challenges.
- Images generated from five prominent TTI models: Stable Diffusion (v1.4, v1.5, v2.0, XL-base) and DALL-E 3.
Key Insights and Ideas
- Prompt Engineering Has Limits: A common assumption is that adding more detail to a prompt fixes generation errors. The authors show that even with highly detailed, iterative prompt refinement, TTI models still struggle to accurately depict complex factual structures like molecular geometry.
- Text Alignment $\neq$ Factuality: Due to polysemy (words with multiple meanings), a model can generate an image that technically matches a prompt phrase but is historically or scientifically wrong. For instance, a prompt for "comb-pattern pottery" might yield a modern pot with a comb design rather than the specific ancient artifact type.
- VLMs as Reliable Evaluators: With the right scaffolding (structured factual reasoning), advanced vision-language models can evaluate abstract concepts like "factuality" almost as accurately as a crowd of human judges.
Key Results
- DALL-E 3 Leads, But Still Hallucinates: Across all domains, DALL-E 3 significantly outperformed the Stable Diffusion models, scoring 0.661 in science and 0.666 in history. However, it still lagged far behind the baseline scores of true textbook images (0.856 and 0.873, respectively), highlighting a major remaining gap in TTI factuality.
- Strong Human Correlation: The authors validated I-HallA by conducting a user study with 53 participants. The automated GPT-4o evaluation achieved a Spearman correlation of 0.950 with human scores, proving it to be a highly dependable proxy for human evaluation.
Limitations and Future Work
The primary limitation acknowledged by the authors is that even true factual images do not score a perfect 1.0 (hovering around 0.85–0.87). This discrepancy is caused by inherent noise in automated QA generation and occasional visual comprehension limitations within the VQA model itself.
Future work will focus on minimizing this baseline noise and exploring tighter integration methods to help TTI models natively reference external knowledge bases during the generation process.
Summary updated