Abstract
Automatic image captioning evaluation is critical for benchmarking and promoting advances in image captioning research. Existing metrics only provide a single score to measure caption qualities, which are less explainable and informative. Instead, we humans can easily identify the problems of captions in details, e.g., which words are inaccurate and which salient objects are not described, and then rate the caption quality. To support such informative feedback, we propose an Informative Metric for Reference-free Image Caption evaluation (InfoMetIC). Given an image and a caption, InfoMetIC is able to report incorrect words and unmentioned image regions at fine-grained level, and also provide a text precision score, a vision recall score and an overall quality score at coarse-grained level. The coarse-grained score of InfoMetIC achieves significantly better correlation with human judgements than existing metrics on multiple benchmarks. We also construct a token-level evaluation dataset and demonstrate the effectiveness of InfoMetIC in fine-grained evaluation.
Introduction
Evaluating the quality of machine-generated image captions is a fundamental challenge in computer vision and natural language processing. Traditional evaluation metrics, such as CIDEr and SPICE, rely heavily on comparing generated captions to a small set of human-written reference captions. This approach suffers from "reference scarcity," where a perfectly valid description of an image may be penalized simply because it does not match the specific wording of the provided references.
Furthermore, existing metrics—including recent reference-free models like CLIP-S—operate as "black boxes." They provide a single numerical score but offer no diagnostic feedback. They cannot explain why a caption is poor, such as identifying specific factual inaccuracies or noting that a model missed a salient object. To bridge this gap, we introduce InfoMetIC, a reference-free metric designed to provide both high-level quality scores and fine-grained, explainable feedback.
The InfoMetIC Framework
InfoMetIC is built to mimic human evaluation by assessing two primary dimensions: Text Precision (is the information in the caption accurate relative to the image?) and Vision Recall (does the caption cover the important visual elements of the image?).
Addressing "Context Overuse"
A key innovation in InfoMetIC is its approach to token encoding. Standard vision-language models often suffer from "context overuse," where the representation of an individual word becomes overly influenced by the surrounding sentence. This makes it difficult to isolate which specific word is incorrect. InfoMetIC addresses this by encoding text tokens independently, preserving their individual semantic meaning before they are aligned with image regions.
Dual-Stream Scoring
The architecture utilizes a cross-modal Transformer to facilitate interaction between visual and textual features through two distinct streams:
- Text Precision Stream: A vision-filtered text encoder identifies which words in the caption are supported by the visual evidence in the image.
- Vision Recall Stream: A text-filtered vision encoder identifies which salient regions of the image have been successfully described in the caption.
Multi-task Learning and Hard Negatives
To ensure the metric is sensitive to subtle errors, InfoMetIC is trained using multi-task learning. This includes coarse-grained contrastive learning and fine-grained sequence labeling. Crucially, the training process incorporates "Hard Textual Negatives"—captions that are mostly correct but contain minor, specific errors—forcing the model to develop high sensitivity to detail.
CapTokenEval: A New Benchmark for Explainability
To evaluate the diagnostic capabilities of InfoMetIC, we introduced CapTokenEval. This dataset consists of 700 image-caption pairs with human annotations at the token level. Annotators marked exactly which words were incorrect and which image regions were mentioned. This provides a gold standard for measuring how well an automated metric can provide fine-grained feedback.
Experimental Results
InfoMetIC demonstrates superior performance across multiple standard benchmarks and our new diagnostic dataset:
- Human Correlation: On the Pascal-50S benchmark, InfoMetIC+ achieved a mean accuracy of 86.5%, outperforming CLIP-S (83.7%) and rivaling many reference-based metrics.
- Diagnostic Accuracy: On the CapTokenEval benchmark, InfoMetIC achieved 0.80 accuracy in identifying incorrect text tokens and 0.75 accuracy in identifying mentioned image regions. This confirms its ability to provide meaningful explanations for its scores.
- Generalization: Testing on the NoCaps dataset showed that InfoMetIC generalizes well to out-of-domain objects not seen during training, indicating it has learned robust visual-textual concepts.
Conclusion
InfoMetIC represents a significant step forward in image caption evaluation by moving beyond single-score metrics toward explainable, reference-free assessment. By providing fine-grained feedback on text precision and vision recall, it offers researchers a powerful tool for diagnosing model weaknesses. Future work will explore extending this framework to specialized domains, such as medical imaging, and expanding its scope to video captioning and text-aware visual tasks.
Summary updated