Quality Estimation for Image Captions Based on Large-scale Human Evaluations

Tomer Levinboim, Ashish V. Thapliyal, Piyush Sharma, Radu Soricut

Abstract: Tomer Levinboim, Ashish V. Thapliyal, Piyush Sharma, Radu Soricot. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

2021.

Introduction

While AI-driven image captioning has advanced significantly, models still frequently "hallucinate" or produce generic, unhelpful descriptions. In high-stakes scenarios—such as providing accessibility for blind or visually impaired (BVI) users—an incorrect caption is more than a minor error; it is a breach of trust that can lead to a fundamental misunderstanding of the environment.

Traditionally, these models are evaluated using metrics like CIDEr or SPICE, which require a human-written "ground-truth" reference for comparison. However, when a model is deployed in real-world applications, no such reference exists. This research addresses the challenge of Quality Estimation (QE): predicting the quality of a caption based solely on the image and the generated text at the time of prediction.

The Bilinear QE Model

The authors propose a Bilinear QE model designed to capture the nuanced relationship between visual content and language through three primary stages:

  1. Feature Extraction: The model utilizes three pre-trained "experts":
  2. Global Image Embedding: A Graph-RISE model that summarizes the entire image into a compact vector.
  3. Object Labels: A Faster R-CNN detector that identifies specific objects and maps them to embeddings.
  4. Caption Embedding: The Universal Sentence Encoder (USE) to represent the generated text.
  5. Bilinear Interactions: Rather than simply concatenating these features, the model employs bilinear layers to explicitly calculate interactions between three specific pairings: (Object, Image), (Object, Caption), and (Image, Caption). This allows the system to verify if objects mentioned in the text are actually present in the visual data.
  6. Training Strategy: The model is "warm-started" on a massive image-text similarity task using the Conceptual Captions dataset before being fine-tuned on a new human-rated dataset.

Data Collection and the Caption-Quality Dataset

To train the model, the researchers introduced the Caption-Quality Dataset, a massive benchmark consisting of roughly 65,000 image-caption pairs and over 600,000 binary human ratings.

The binary rating interface used to collect over 600,000 human judgments.

A key insight from the study is that "simplicity scales." Instead of complex rubrics, a simple binary question—"Is this a good caption?"—aggregated across 10 different raters provides a nuanced probability score that serves as a highly effective training signal. Stability analysis showed an 85% agreement rate among different groups of raters, confirming the reliability of this crowdsourced data.

Fine-grained Evaluation: Helpfulness and Correctness

Beyond binary quality, the researchers developed Caption-Ext, a smaller dataset where professional raters evaluated captions based on two critical dimensions: Helpfulness (does it provide useful info for someone who cannot see the image?) and Correctness (is the information accurate?).

The fine-grained rating interface for the Caption-Ext dataset, measuring helpfulness and correctness.

Key Results and Impact

The fine-tuned bilinear model achieved a Spearman’s ρ of 0.53 on the test set, significantly outperforming standard image-text similarity baselines (0.25). More importantly, the QE model demonstrated immense filtering power. In extrinsic evaluations, to maintain a precision level of 0.8 (where 80% of shown captions are both correct and helpful):

  • A standard model could only safely serve 21% of its generated captions.
  • The proposed QE model could serve 71% of its captions—a 3.4x improvement in the utility of the system.

Conclusion and Future Directions

While the model is highly effective at identifying low-quality captions, it currently operates as a "black box" that does not explain why a caption is poor (e.g., missing objects vs. incorrect colors). Additionally, its performance on specialized domains like medical imagery remains to be tested.

Future work suggests using these QE scores as a reward signal in Reinforcement Learning (RL). This would allow captioning models to become "self-aware" during training, inherently reducing the likelihood of producing untrustworthy outputs.

Summary updated

BibTeX
@inproceedings{levinboim2021quality,
  author = {Tomer Levinboim and Ashish V. Thapliyal and Piyush Sharma and Radu Soricut},
  title = {{Quality Estimation for Image Captions Based on Large-scale Human Evaluations}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=nk93Zexs-Mz},
}
Figure