Quality Estimation for Image Captions Based on Large-scale Human Evaluations

Tomer Levinboim, Ashish V. Thapliyal, Piyush Sharma, Radu Soricut

Abstract

Automatic image captioning has improved significantly over the last few years, but the problem is far from being solved, with state of the art models still often producing low quality captions when used in the wild. In this paper, we focus on the task of Quality Estimation (QE) for image captions, which attempts to model the caption quality from a human perspective and without access to ground-truth references, so that it can be applied at prediction time to detect low-quality captions produced on previously unseen images. For this task, we develop a human evaluation process that collects coarse-grained caption annotations from crowdsourced users, which is then used to collect a large scale dataset spanning more than 600k caption quality ratings. We then carefully validate the quality of the collected ratings and establish baseline models for this new QE task. Finally, we further collect fine-grained caption quality annotations from trained raters, and use them to demonstrate that QE models trained over the coarse ratings can effectively detect and filter out low-quality image captions, thereby improving the user experience from captioning systems.

Introduction

While automatic image captioning has made significant strides, models deployed in real-world scenarios frequently "hallucinate" objects or fail to capture the essence of an image. In research settings, these models are typically evaluated using metrics like CIDEr or SPICE, which require human-written reference captions for comparison. However, in practical applications—such as assistive technology for the visually impaired—reference captions are unavailable. If a model incorrectly describes a scene, the user has no way to verify the output. This paper addresses the challenge of Reference-Free Quality Estimation (QE): developing a "critic" model that evaluates an image and its generated caption to predict human-perceived quality without a ground truth.

Methodology and Data Collection

The authors propose a two-pronged approach involving large-scale data collection and a specialized model architecture.

Data Collection Strategy

To build a robust QE model, the researchers created the Caption-Quality dataset. This dataset contains approximately 65,000 image-caption pairs from the Open Images Dataset (OID), supported by 600,000 human judgments.

The core innovation in data collection was the use of a "Fast & Simple" binary rating system. Instead of complex linguistic analysis, crowdsourced workers were asked a single question: "Is this a good caption for the image?" By aggregating 8–10 of these binary votes per pair, the authors derived a stable, continuous quality score ($\hat{p}$).

Crowdsourcing interface for binary caption quality ratings.

To validate these coarse ratings, a smaller Caption-Ext dataset was created. Professional raters provided fine-grained evaluations across dimensions such as "helpfulness" and "correctness," ensuring the binary signals correlated with deeper utility.

Fine-grained evaluation interface for professional raters.

The Bilinear QE Model

The proposed neural network architecture processes three distinct inputs:

  1. Global Image Embeddings: High-level visual features extracted via Graph-RISE.
  2. Object Labels: Specific scene objects identified by Faster R-CNN.
  3. Caption Embeddings: Textual representations generated by the Universal Sentence Encoder.

A bilinear layer is employed to model the interactions between these visual and textual features, specifically checking if detected objects align with the caption's content. The model was pre-trained on the Conceptual Captions dataset before being fine-tuned on the human-labeled quality ratings.

Key Results and Insights

The research yielded several critical insights into the nature of caption quality:

  • Similarity vs. Quality: Models trained solely on image-text similarity (a common proxy) perform poorly at predicting human quality ratings. The study found that human-in-the-loop data is essential to teach models what constitutes a "good" caption.
  • Performance Metrics: The best bilinear model achieved a Spearman’s correlation of 0.53 with human judgments, more than doubling the performance of baseline similarity models (0.25).
  • Practical Utility: The QE model serves as an effective gatekeeper. To maintain 80% precision (where 8 out of 10 shown captions are high quality), the QE-filtered system could serve 71% of all generated captions. In contrast, a standard similarity-based model could only serve 21% while maintaining the same quality threshold.

Conclusion

This work demonstrates that simple, binary human feedback can be scaled to create highly effective quality estimation models. By acting as a filter, these models can significantly improve the reliability of AI-generated content for end-users. Future research will explore using QE scores as reward signals in reinforcement learning to improve caption generators directly and expanding the framework to provide granular feedback on specific errors.

Summary updated

Abstract

Automatic image captioning has improved significantly over the last few years, but the problem is far from being solved, with state of the art models still often producing low quality captions when used in the wild. In this paper, we focus on the task of Quality Estimation (QE) for image captions, which attempts to model the caption quality from a human perspective and without access to ground-truth references, so that it can be applied at prediction time to detect low-quality captions produced on previously unseen images. For this task, we develop a human evaluation process that collects coarse-grained caption annotations from crowdsourced users, which is then used to collect a large scale dataset spanning more than 600k caption quality ratings. We then carefully validate the quality of the collected ratings and establish baseline models for this new QE task. Finally, we further collect fine-grained caption quality annotations from trained raters, and use them to demonstrate that QE models trained over the coarse ratings can effectively detect and filter out low-quality image captions, thereby improving the user experience from captioning systems.
BibTeX
@inproceedings{levinboim2021quality,
  author = {Tomer Levinboim and Ashish V. Thapliyal and Piyush Sharma and Radu Soricut},
  title = {{Quality Estimation for Image Captions Based on Large-scale Human Evaluations}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=nk93Zexs-Mz},
}
Figure