Introduction
Evaluating machine-generated image captions is a complex challenge. Traditional reference-based metrics like BLEU, CIDEr, and SPICE rely on n-gram overlaps with human-written "ground truth" references. This approach suffers from semantic blindness and reference scarcity, often penalizing valid descriptions that simply differ from the provided references. While recent reference-free metrics like CLIP-S address these issues by comparing captions directly to images, they operate as "black boxes," providing a single quality score without diagnostic insight.
InfoMetIC (Informative Metric for Image Captioning) bridges this gap by providing fine-grained, explainable feedback. It decomposes evaluation into two intuitive axes: text precision (accuracy of the words used) and vision recall (completeness of the description relative to the image).
Methodology
InfoMetIC is built upon the CLIP (ViT-B/32) architecture and comprises three primary modules designed to capture fine-grained cross-modal relationships.
Architecture Overview
- Token-level Encoding: To prevent "context overuse"—where a model ignores specific word errors due to the overall sentence flow—InfoMetIC encodes text tokens independently. Simultaneously, an object detector extracts features for specific image regions.
- Intra & Inter Modality Fusion: Multi-layer Transformers model relationships within each modality (e.g., spatial relations between image regions) and across modalities (e.g., grounding words to specific regions).
- Fine-grained Scoring Module: This module utilizes cross-modal attention to calculate the two core components of the metric:
- Text Precision: The image is used to "filter" the text, identifying tokens that do not correspond to visual content (hallucinations).
- Vision Recall: The text is used to "filter" the image, identifying salient regions that the caption failed to describe.
Training Objectives
The model is trained using a multi-task approach. Coarse-grained Score Learning employs contrastive loss with "hard" textual negatives—captions that are similar but more detailed than the target—to sharpen the model's vision recall. Fine-grained Score Learning is framed as a sequence labeling task, training the model to detect "polluted" or incorrect words within a sentence.
Experimental Results
The performance of InfoMetIC was validated across several established benchmarks and a newly introduced diagnostic dataset.
Correlation with Human Judgment
InfoMetIC achieves state-of-the-art correlation with human judgments on the Composite and Pascal-50S datasets. When combined with standard CLIP similarity (InfoMetIC+), the metric reaches 86.5% accuracy on the Pascal-50S benchmark, outperforming existing reference-free alternatives.
Diagnostic Capabilities
To evaluate explainability, the authors introduced CapTokenEval, a benchmark of 700 image-caption pairs manually annotated for token-level errors and region grounding. InfoMetIC demonstrated high diagnostic accuracy, achieving 0.80 in identifying incorrect text tokens and 0.75 for vision regions.
Generalization
Evaluations on the NoCaps dataset indicate strong zero-shot generalization, showing that InfoMetIC can effectively evaluate descriptions of out-of-domain objects not seen during training.
Discussion and Conclusion
InfoMetIC represents a significant shift in caption evaluation by prioritizing explainability. By explicitly modeling precision and recall, it aligns more closely with human intuition than monolithic similarity scores. The use of independent token encoding successfully mitigates context overuse, allowing for the detection of subtle errors that global embedding models often miss.
While highly effective, future work remains to address domain gaps in specialized fields (e.g., medical imaging) and to incorporate explicit handling of scene text. The authors also plan to extend this framework to video captioning and text-aware image captioning tasks. In summary, InfoMetIC provides a robust, diagnostic tool for developers to pinpoint exactly where their image captioning models succeed or fail.
Summary updated