Abstract
Vision-language models (VLMs) rely on high-quality image-caption pairs, yet web-scale datasets often contain significant visual-textual inconsistencies (VTI). Traditional metrics like CLIPScore fail to capture fine-grained errors or handle complex, long captions. We present HMGIE (Hierarchical and Multi-Grained Inconsistency Evaluation), a framework that decomposes captions into structured semantic graphs and employs a progressive, multi-level question-answering strategy. HMGIE provides nuanced assessments of both semantic accuracy ($H_{acc}$) and completeness ($H_{comp}$), significantly outperforming existing methods on the new MVTID benchmark and demonstrating superior alignment with human judgment.
Introduction
The performance of state-of-the-art vision-language models is fundamentally tied to the quality of their training data. However, massive datasets are frequently riddled with inconsistencies where captions misdescribe visual elements. Existing evaluation tools suffer from "granularity blindness"—missing subtle errors like misidentifying an SUV as a truck—and "complexity collapse," where they fail to verify every detail in long, descriptive captions.
To address these issues, we introduce the Multi-granularity Visual-Textual Inconsistency Dataset (MVTID), which categorizes captions into four levels of detail: Basic, Extended, Detailed, and Complete. This dataset serves as a rigorous benchmark for evaluating fine-grained consistency.
Methods
HMGIE operates through a three-module pipeline designed to move beyond global embedding similarities:
- Semantic Graph Generation: An LLM parses the caption into a Semantic Graph ($G_{sg}$), where nodes represent entities, attributes, and events, and edges represent their relationships. This serves as a structural map for the evaluation.
- Hierarchical Inconsistency Evaluation: The system constructs a Hierarchical Inconsistency Evaluation Graph (HIEG) progressively. It verifies basic objects first, followed by attributes and complex spatial arrangements. A semantic coverage mask ensures every part of the original caption is checked via Visual Question Answering (VQA).
- Quantitative Evaluation: The framework outputs two key metrics: $H_{acc}$ (Accuracy), measuring the correctness of claims, and $H_{comp}$ (Completeness), measuring visual scene coverage.
Results
HMGIE achieves state-of-the-art performance on the MVTID dataset with an F1 score of 92.68%, a 22% improvement over previous methods. Key insights include:
- Decomposition Effectiveness: Node-by-node verification prevents the "bag-of-words" effect, ensuring specific attributes are not ignored.
- Decoupling Accuracy and Completeness: HMGIE distinguishes between a short, accurate caption and a long, detailed one that might contain a single error.
- Robustness to Length: While standard LLM prompting performance drops significantly as captions lengthen, HMGIE remains stable, with only a ~6% performance decrease from basic to highly detailed captions.
- Human Alignment: HMGIE’s $H_{acc}$ score shows a significantly higher correlation with human judgment compared to CLIPScore, particularly for detailed descriptions.
Conclusion
HMGIE offers a robust solution for vision-language data cleansing. While computationally more intensive than one-shot methods, its superior accuracy and human alignment make it a powerful tool for improving VLM training data. Future work will focus on model distillation for efficiency and integrating external knowledge for complex tasks like news verification.
Summary updated