HMGIE: Hierarchical and Multi-Grained Inconsistency Evaluation for Vision-Language Data Cleansing

Zihao Zhu, Hongbao Zhang, Guanzong Wu, Siwei Lyu, Baoyuan WuarXiv 2024 · DOI · arXiv

Abstract

Vision-language models (VLMs) rely on high-quality image-caption pairs, yet web-scale datasets often contain significant visual-textual inconsistencies (VTI). Traditional metrics like CLIPScore fail to capture fine-grained errors or handle complex, long captions. We present HMGIE (Hierarchical and Multi-Grained Inconsistency Evaluation), a framework that decomposes captions into structured semantic graphs and employs a progressive, multi-level question-answering strategy. HMGIE provides nuanced assessments of both semantic accuracy ($H_{acc}$) and completeness ($H_{comp}$), significantly outperforming existing methods on the new MVTID benchmark and demonstrating superior alignment with human judgment.

Introduction

The performance of state-of-the-art vision-language models is fundamentally tied to the quality of their training data. However, massive datasets are frequently riddled with inconsistencies where captions misdescribe visual elements. Existing evaluation tools suffer from "granularity blindness"—missing subtle errors like misidentifying an SUV as a truck—and "complexity collapse," where they fail to verify every detail in long, descriptive captions.

Comparison of HMGIE with existing metrics across different caption granularities for a sample image.

To address these issues, we introduce the Multi-granularity Visual-Textual Inconsistency Dataset (MVTID), which categorizes captions into four levels of detail: Basic, Extended, Detailed, and Complete. This dataset serves as a rigorous benchmark for evaluating fine-grained consistency.

Methods

HMGIE operates through a three-module pipeline designed to move beyond global embedding similarities:

  1. Semantic Graph Generation: An LLM parses the caption into a Semantic Graph ($G_{sg}$), where nodes represent entities, attributes, and events, and edges represent their relationships. This serves as a structural map for the evaluation.
  2. Hierarchical Inconsistency Evaluation: The system constructs a Hierarchical Inconsistency Evaluation Graph (HIEG) progressively. It verifies basic objects first, followed by attributes and complex spatial arrangements. A semantic coverage mask ensures every part of the original caption is checked via Visual Question Answering (VQA).
  3. Quantitative Evaluation: The framework outputs two key metrics: $H_{acc}$ (Accuracy), measuring the correctness of claims, and $H_{comp}$ (Completeness), measuring visual scene coverage.
The HMGIE pipeline, featuring Semantic Graph Generation and the Hierarchical Inconsistency Evaluation Module.

Results

HMGIE achieves state-of-the-art performance on the MVTID dataset with an F1 score of 92.68%, a 22% improvement over previous methods. Key insights include:

  • Decomposition Effectiveness: Node-by-node verification prevents the "bag-of-words" effect, ensuring specific attributes are not ignored.
  • Decoupling Accuracy and Completeness: HMGIE distinguishes between a short, accurate caption and a long, detailed one that might contain a single error.
  • Robustness to Length: While standard LLM prompting performance drops significantly as captions lengthen, HMGIE remains stable, with only a ~6% performance decrease from basic to highly detailed captions.
  • Human Alignment: HMGIE’s $H_{acc}$ score shows a significantly higher correlation with human judgment compared to CLIPScore, particularly for detailed descriptions.

Conclusion

HMGIE offers a robust solution for vision-language data cleansing. While computationally more intensive than one-shot methods, its superior accuracy and human alignment make it a powerful tool for improving VLM training data. Future work will focus on model distillation for efficiency and integrating external knowledge for complex tasks like news verification.

Summary updated

Abstract

Visual-textual inconsistency (VTI) evaluation plays a crucial role in cleansing vision-language data. Its main challenges stem from the high variety of image captioning datasets, where differences in content can create a range of inconsistencies (\eg, inconsistencies in scene, entities, entity attributes, entity numbers, entity interactions). Moreover, variations in caption length can introduce inconsistencies at different levels of granularity as well. To tackle these challenges, we design an adaptive evaluation framework, called Hierarchical and Multi-Grained Inconsistency Evaluation (HMGIE), which can provide multi-grained evaluations covering both accuracy and completeness for various image-caption pairs. Specifically, the HMGIE framework is implemented by three consecutive modules. Firstly, the semantic graph generation module converts the image caption to a semantic graph for building a structural representation of all involved semantic items. Then, the hierarchical inconsistency evaluation module provides a progressive evaluation procedure with a dynamic question-answer generation and evaluation strategy guided by the semantic graph, producing a hierarchical inconsistency evaluation graph (HIEG). Finally, the quantitative evaluation module calculates the accuracy and completeness scores based on the HIEG, followed by a natural language explanation about the detection results. Moreover, to verify the efficacy and flexibility of the proposed framework on handling different image captioning datasets, we construct MVTID, an image-caption dataset with diverse types and granularities of inconsistencies. Extensive experiments on MVTID and other benchmark datasets demonstrate the superior performance of the proposed HMGIE to current state-of-the-art methods.
BibTeX
@misc{zhu2024hmgie,
  author = {Zihao Zhu and Hongbao Zhang and Guanzong Wu and Siwei Lyu and Baoyuan Wu},
  title = {{HMGIE: Hierarchical and Multi-Grained Inconsistency Evaluation for Vision-Language Data Cleansing}},
  year = {2024},
  eprint = {2412.05685},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.05685},
}
Figure