REO-Relevance, Extraness, Omission: A Fine-grained Evaluation for Image Captioning

Ming Jiang, Junjie Hu, Qiuyuan Huang, Lei Zhang, Jana Diesner, Jianfeng Gao

Abstract

Evaluating image captioning systems is challenging because traditional composite metrics like BLEU, CIDEr, and SPICE offer a single numerical score without explaining the nature of errors. We introduce REO (Relevance, Extraness, Omission), a fine-grained evaluation framework that decomposes caption quality into three interpretable components. By leveraging neural grounding through a Stacked Cross Attention Network (SCAN) and orthogonal vector projection, REO identifies specific failures such as hallucinations (Extraness) and missing details (Omission). Our experiments show that REO significantly outperforms existing metrics in correlating with human judgment across multiple benchmarks, providing a diagnostic tool for targeted model improvement.

Introduction: The Need for Diagnostic Metrics

The current landscape of image captioning evaluation relies heavily on "black box" metrics. While these metrics can rank models, they fail to provide actionable feedback. If a model receives a low score, developers cannot easily discern if the caption is factually incorrect, missing the primary subject, or hallucinating non-existent objects. This lack of interpretability hinders the debugging process. REO addresses this gap by providing a metric that explains how a caption deviates from the ground truth.

Methodology: The REO Framework

The REO framework decomposes evaluation into three pillars:

  1. Relevance (R): Measures the semantic alignment between the candidate caption and the ground truth.
  2. Extraness (E): Identifies irrelevant information or "hallucinations" present in the candidate but absent in the ground truth (False Positives).
  3. Omission (O): Detects elements present in the image or reference captions that the candidate failed to describe (False Negatives).

Technical Implementation via Orthogonal Projection

To calculate these scores, REO utilizes a pre-trained Stacked Cross Attention Network (SCAN), which maps image regions and caption words into a shared multi-modal semantic space.

The REO framework architecture using the SCAN model and orthogonal vector projection.

The core innovation lies in the use of orthogonal vector projection. While Relevance is determined by standard cosine similarity, Extraness and Omission are calculated by isolating the components of the candidate vector that are perpendicular to the ground truth vector. This "leftover" vector represents information present in one but missing in the other. A unique strength of REO is its ability to use both the original image and human-written references as ground truth, offering a more holistic evaluation than text-only metrics.

Experimental Setup and Datasets

The effectiveness of REO was validated across three major benchmarks:

  • Composite Dataset: Approximately 12,000 captions from MS-COCO and Flickr datasets with human correctness ratings.
  • PublicSys Dataset: 2,500 captions generated by five state-of-the-art systems on MS-COCO images.
  • Pascal-50S: 4,000 caption pairs used to evaluate the metric's ability to match human preferences in pairwise comparisons.

Key Results and Insights

REO consistently outperformed traditional metrics across all tested datasets. On the Composite dataset, REO (using both Image and Reference) achieved a Kendall tau ($\tau$) of 0.533, nearly doubling the performance of BLEU-1 (0.280) and significantly exceeding SPICE (0.419).

Examples of REO identifying specific errors like Omission (green) and Extraness (red).

Diagnostic Capabilities

A key insight from the study is that human annotators are more sensitive to "Extraness" and "Omission" than to general "Relevance." We penalize a model more for hallucinating a "cat" than we reward it for correctly identifying a "dog." REO captures this nuance. Manual validation of the "error vectors" identified by the model showed a high similarity ($\ge 0.65$) to actual human-identified errors, confirming that the metric accurately captures specific semantic failures.

Conclusion and Future Work

REO provides a significant step forward in making image captioning evaluation interpretable. By mathematically "subtracting" correct information to isolate errors, it offers researchers a diagnostic tool for targeted debugging.

However, the framework's performance is dependent on the underlying grounding model (SCAN). Future work could involve integrating more advanced multi-modal models to improve accuracy. Additionally, while currently focused on image captioning, the principles of REO could be extended to other tasks such as video captioning or document summarization.

Summary updated

BibTeX
@inproceedings{jiang2019reo,
  author = {Ming Jiang and Junjie Hu and Qiuyuan Huang and Lei Zhang and Jana Diesner and Jianfeng Gao},
  title = {{REO-Relevance, Extraness, Omission: A Fine-grained Evaluation for Image Captioning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-1156},
}
Figure