InfoMetIC: An Informative Metric for Reference-free Image Caption Evaluation

Anwen Hu, Shizhe Chen, Liang Zhang, Qin Jin

Introduction

Evaluating machine-generated image captions is a complex challenge. Traditional reference-based metrics like BLEU, CIDEr, and SPICE rely on n-gram overlaps with human-written "ground truth" references. This approach suffers from semantic blindness and reference scarcity, often penalizing valid descriptions that simply differ from the provided references. While recent reference-free metrics like CLIP-S address these issues by comparing captions directly to images, they operate as "black boxes," providing a single quality score without diagnostic insight.

InfoMetIC (Informative Metric for Image Captioning) bridges this gap by providing fine-grained, explainable feedback. It decomposes evaluation into two intuitive axes: text precision (accuracy of the words used) and vision recall (completeness of the description relative to the image).

InfoMetIC provides explainable feedback by identifying specific word errors and unmentioned image regions, unlike traditional metrics.

Methodology

InfoMetIC is built upon the CLIP (ViT-B/32) architecture and comprises three primary modules designed to capture fine-grained cross-modal relationships.

Architecture Overview

  1. Token-level Encoding: To prevent "context overuse"—where a model ignores specific word errors due to the overall sentence flow—InfoMetIC encodes text tokens independently. Simultaneously, an object detector extracts features for specific image regions.
  2. Intra & Inter Modality Fusion: Multi-layer Transformers model relationships within each modality (e.g., spatial relations between image regions) and across modalities (e.g., grounding words to specific regions).
  3. Fine-grained Scoring Module: This module utilizes cross-modal attention to calculate the two core components of the metric:
  4. Text Precision: The image is used to "filter" the text, identifying tokens that do not correspond to visual content (hallucinations).
  5. Vision Recall: The text is used to "filter" the image, identifying salient regions that the caption failed to describe.
The InfoMetIC architecture uses separate encoders and a fine-grained scoring module to compute precision and recall.

Training Objectives

The model is trained using a multi-task approach. Coarse-grained Score Learning employs contrastive loss with "hard" textual negatives—captions that are similar but more detailed than the target—to sharpen the model's vision recall. Fine-grained Score Learning is framed as a sequence labeling task, training the model to detect "polluted" or incorrect words within a sentence.

Experimental Results

The performance of InfoMetIC was validated across several established benchmarks and a newly introduced diagnostic dataset.

Correlation with Human Judgment

InfoMetIC achieves state-of-the-art correlation with human judgments on the Composite and Pascal-50S datasets. When combined with standard CLIP similarity (InfoMetIC+), the metric reaches 86.5% accuracy on the Pascal-50S benchmark, outperforming existing reference-free alternatives.

Diagnostic Capabilities

To evaluate explainability, the authors introduced CapTokenEval, a benchmark of 700 image-caption pairs manually annotated for token-level errors and region grounding. InfoMetIC demonstrated high diagnostic accuracy, achieving 0.80 in identifying incorrect text tokens and 0.75 for vision regions.

Generalization

Evaluations on the NoCaps dataset indicate strong zero-shot generalization, showing that InfoMetIC can effectively evaluate descriptions of out-of-domain objects not seen during training.

Discussion and Conclusion

InfoMetIC represents a significant shift in caption evaluation by prioritizing explainability. By explicitly modeling precision and recall, it aligns more closely with human intuition than monolithic similarity scores. The use of independent token encoding successfully mitigates context overuse, allowing for the detection of subtle errors that global embedding models often miss.

While highly effective, future work remains to address domain gaps in specialized fields (e.g., medical imaging) and to incorporate explicit handling of scene text. The authors also plan to extend this framework to video captioning and text-aware image captioning tasks. In summary, InfoMetIC provides a robust, diagnostic tool for developers to pinpoint exactly where their image captioning models succeed or fail.

Summary updated

Abstract

Automatic image captioning evaluation is critical for benchmarking and promoting advances in image captioning research. Existing metrics only provide a single score to measure caption qualities, which are less explainable and informative. Instead, we humans can easily identify the problems of captions in details, e.g., which words are inaccurate and which salient objects are not described, and then rate the caption quality. To support such informative feedback, we propose an Informative Metric for Reference-free Image Caption evaluation (InfoMetIC). Given an image and a caption, InfoMetIC is able to report incorrect words and unmentioned image regions at fine-grained level, and also provide a text precision score, a vision recall score and an overall quality score at coarse-grained level. The coarse-grained score of InfoMetIC achieves significantly better correlation with human judgements than existing metrics on multiple benchmarks. We also construct a token-level evaluation dataset and demonstrate the effectiveness of InfoMetIC in fine-grained evaluation. Our code and datasets are publicly available at https://github.com/HAWLYQ/InfoMetIC.
BibTeX
@inproceedings{hu2023infometic,
  author = {Anwen Hu and Shizhe Chen and Liang Zhang and Qin Jin},
  title = {{InfoMetIC: An Informative Metric for Reference-free Image Caption Evaluation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  url = {https://openreview.net/forum?id=zEjfOcgPMyM},
}
Figure