Abstract: Tomer Levinboim, Ashish V. Thapliyal, Piyush Sharma, Radu Soricot. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
2021.
Introduction
While AI-driven image captioning has advanced significantly, models still frequently "hallucinate" or produce generic, unhelpful descriptions. In high-stakes scenarios—such as providing accessibility for blind or visually impaired (BVI) users—an incorrect caption is more than a minor error; it is a breach of trust that can lead to a fundamental misunderstanding of the environment.
Traditionally, these models are evaluated using metrics like CIDEr or SPICE, which require a human-written "ground-truth" reference for comparison. However, when a model is deployed in real-world applications, no such reference exists. This research addresses the challenge of Quality Estimation (QE): predicting the quality of a caption based solely on the image and the generated text at the time of prediction.
The Bilinear QE Model
The authors propose a Bilinear QE model designed to capture the nuanced relationship between visual content and language through three primary stages:
- Feature Extraction: The model utilizes three pre-trained "experts":
- Global Image Embedding: A Graph-RISE model that summarizes the entire image into a compact vector.
- Object Labels: A Faster R-CNN detector that identifies specific objects and maps them to embeddings.
- Caption Embedding: The Universal Sentence Encoder (USE) to represent the generated text.
- Bilinear Interactions: Rather than simply concatenating these features, the model employs bilinear layers to explicitly calculate interactions between three specific pairings: (Object, Image), (Object, Caption), and (Image, Caption). This allows the system to verify if objects mentioned in the text are actually present in the visual data.
- Training Strategy: The model is "warm-started" on a massive image-text similarity task using the Conceptual Captions dataset before being fine-tuned on a new human-rated dataset.
Data Collection and the Caption-Quality Dataset
To train the model, the researchers introduced the Caption-Quality Dataset, a massive benchmark consisting of roughly 65,000 image-caption pairs and over 600,000 binary human ratings.
A key insight from the study is that "simplicity scales." Instead of complex rubrics, a simple binary question—"Is this a good caption?"—aggregated across 10 different raters provides a nuanced probability score that serves as a highly effective training signal. Stability analysis showed an 85% agreement rate among different groups of raters, confirming the reliability of this crowdsourced data.
Fine-grained Evaluation: Helpfulness and Correctness
Beyond binary quality, the researchers developed Caption-Ext, a smaller dataset where professional raters evaluated captions based on two critical dimensions: Helpfulness (does it provide useful info for someone who cannot see the image?) and Correctness (is the information accurate?).
Key Results and Impact
The fine-tuned bilinear model achieved a Spearman’s ρ of 0.53 on the test set, significantly outperforming standard image-text similarity baselines (0.25). More importantly, the QE model demonstrated immense filtering power. In extrinsic evaluations, to maintain a precision level of 0.8 (where 80% of shown captions are both correct and helpful):
- A standard model could only safely serve 21% of its generated captions.
- The proposed QE model could serve 71% of its captions—a 3.4x improvement in the utility of the system.
Conclusion and Future Directions
While the model is highly effective at identifying low-quality captions, it currently operates as a "black box" that does not explain why a caption is poor (e.g., missing objects vs. incorrect colors). Additionally, its performance on specialized domains like medical imagery remains to be tested.
Future work suggests using these QE scores as a reward signal in Reinforcement Learning (RL). This would allow captioning models to become "self-aware" during training, inherently reducing the likelihood of producing untrustworthy outputs.
Summary updated