Abstract
Assessing image-text alignment models such as CLIP is crucial for bridging visual and linguistic representations. Yet existing benchmarks rely on rule-based perturbations or short captions, limiting their ability to measure fine-grained alignment. We introduce AlignBench, a benchmark that provides a new indicator of image-text alignment by evaluating detailed image-caption pairs generated by diverse image-to-text and text-to-image models. Each sentence is annotated for correctness, enabling direct assessment of VLMs as alignment evaluators. Benchmarking a wide range of decoder-based VLMs reveals three key findings: (i) CLIP-based models, even those tailored for compositional reasoning, remain nearly blind; (ii) detectors systematically over-score early sentences; and (iii) they show strong self-preference, favoring their own outputs and harming detection performance.
Introduction
As Vision-Language Models (VLMs) like GPT-4o and Llama-3 advance, they are increasingly used to generate detailed captions or synthesize images from complex prompts. However, these models frequently "hallucinate"—describing non-existent objects or misrepresenting fine-grained details such as spatial orientation. Existing alignment benchmarks often rely on simple rule-based perturbations (e.g., swapping words), which fail to capture the subtle errors present in natural, model-generated text.
AlignBench addresses this gap by providing a rigorous framework to measure fine-grained alignment. By evaluating how well VLMs can detect mismatches in detailed image-caption pairs, the benchmark exposes critical weaknesses in current architectures, including "self-preference" biases and spatial "blindness."
Methods and Dataset Construction
AlignBench utilizes a "model-in-the-loop" approach to generate a large-scale, diverse dataset. The construction process involved:
- Generation: Six state-of-the-art Image-to-Text (I2T) models (e.g., GPT-4o, LLaVA) and two Text-to-Image (T2I) models (e.g., Stable Diffusion 3.5) generated detailed pairs for 2,000 diverse images.
- Annotation: A two-stage human annotation process (crowd-workers followed by expert review) labeled 89,000 sentences as Correct, Incorrect, or Unknown.
- Categorization: Errors were classified into nine fine-grained types: Attribute, Object, Number, Location, Illusion, Direction, Text, Relation, and Other.
The resulting dataset is significantly more complex than previous benchmarks, featuring 89k sentences and a vocabulary of 42k unique words across 50 domains, including real-world images, illustrations, and synthetic generations.
Key Insights and Results
Benchmarking various VLMs as alignment evaluators revealed several critical findings:
- CLIP Blindness: Standard CLIP-based models performed near chance levels (AUROC ~50), proving ineffective at detecting the fine-grained hallucinations produced by modern generative models.
- Self-Preference Bias: Models exhibit a "narcissistic" tendency, being significantly more likely to label a caption as correct if it was generated by the same model, which compromises their objectivity as detectors.
- Positional Bias: Evaluators consistently assign higher correctness scores to the first sentence of a caption, regardless of its actual accuracy.
- Spatial and Numerical Weakness: Models struggle most with Direction and Number categories, indicating a lack of precise spatial reasoning.
While proprietary models like GPT-5 and large open-source models like Llama-4 (400B) are the strongest evaluators, their performance on AlignBench is notably lower than on simpler datasets like FOIL, highlighting the benchmark's difficulty. Furthermore, localization tasks—pinpointing exactly where an error occurs—remain extremely challenging, with top models achieving only 24.2% mIoU.
Conclusion
AlignBench reveals that as generative models improve, their errors become more subtle and difficult to detect. Future work should focus on mitigating self-preference and positional biases, perhaps through Chain-of-Thought reasoning or model ensembling, to build more reliable evaluators for fine-grained image-text alignment.
Summary updated