Informative Image Captioning with External Sources of Information

Sanqiang Zhao, Piyush Sharma, Tomer Levinboim, Radu Soricut

Abstract

An image caption should fluently present the essential information in a given image, including informative, fine-grained entity mentions and the manner in which these entities interact. However, current captioning models are usually trained to generate captions that only contain common object names, thus falling short on an important "informativeness" dimension. We present a mechanism for integrating image information together with fine-grained labels (assumed to be generated by some upstream models) into a caption that describes the image in a fluent and informative manner. We introduce a multimodal, multi-encoder model based on Transformer that ingests both image features and multiple sources of entity labels. We demonstrate that we can learn to control the appearance of these entity labels in the output, resulting in captions that are both fluent and informative.

The Challenge of Informative Captioning

Standard image captioning systems often default to generic, "safe" descriptions. For instance, instead of identifying "Eric Clapton at the Crossroads Guitar Festival," a standard model might simply output "a man playing guitar at a concert." This research identifies two primary hurdles to achieving informativeness:

  1. The Recognition Gap: It is computationally difficult for a single model to simultaneously learn general scene description and the recognition of millions of specific, fine-grained entities.
  2. The Generalization Trade-off: Models are often penalized during training for being specific if the ground truth is generic, leading them to favor hypernyms (e.g., "building" instead of "Empire State Building") to maximize likelihood scores.

Methodology

Multi-Encoder Architecture

The proposed model treats image captioning as an "image + entities-to-caption" task. It utilizes a multi-encoder Transformer architecture with three distinct input streams:

  • Image Encoder: A standard CNN extracts visual features.
  • Object Label Encoder: Processes common object labels (e.g., "guitar") provided by upstream models.
  • Web Entity Label Encoder: Processes fine-grained entities (e.g., specific celebrity names) found via web-based recognition APIs.

A multi-gated decoder features a "soft switch" gating mechanism that adaptively weights the importance of the image, object labels, and web entities at each generation step.

Multi-encoder Transformer architecture for informative captioning.

Label Coverage Control

A key innovation of this work is the introduction of "coverage control" knobs. These boost factors ($W_{we}$ and $W_{obj}$) can be adjusted at inference time to transition from generic to highly detailed descriptions without retraining. The model uses a precision-like score for objects to prevent redundancy and a recall-like score for web entities to encourage the inclusion of relevant specific names.

Label coverage control mechanism showing the integration of boost factors.

Datasets and Training

The researchers utilized a "re-informed" version of the Conceptual Captions dataset (3.3 million pairs). By using original image URLs and Alt-text, they mapped generic hypernyms back to specific entities. The Google Cloud Vision API served as the upstream source for generating the object and web entity labels used as model inputs.

Key Results and Insights

  • Human Evaluation: The proposed model was judged 24.33% more informative and 7.79% more correct than previous baselines, maintaining high language fluency.
  • Quantitative Performance: The model achieved a CIDEr score of 87.25 when using surface-form names, a significant improvement over image-only baselines (51-62).
  • Controllable Detail: The authors found that predicting entity "Types" (e.g., <ARTIST>) and post-processing them to insert names resulted in more fluent captions than direct name generation, which occasionally produced hallucinations.

Conclusion

By integrating external entity labels and employing a novel coverage control mechanism, this model successfully bridges the gap between generic scene description and informative, fine-grained captioning. Future work will focus on reducing dependency on upstream labeler quality and refining the placement of entities within generated text.

Summary updated

Abstract

An image caption should fluently present the essential information in a given image, including informative, fine-grained entity mentions and the manner in which these entities interact. However, current captioning models are usually trained to generate captions that only contain common object names, thus falling short on an important "informativeness" dimension. We present a mechanism for integrating image information together with fine-grained labels (assumed to be generated by some upstream models) into a caption that describes the image in a fluent and informative manner. We introduce a multimodal, multi-encoder model based on Transformer that ingests both image features and multiple sources of entity labels. We demonstrate that we can learn to control the appearance of these entity labels in the output, resulting in captions that are both fluent and informative.
BibTeX
@inproceedings{zhao2019informative,
  author = {Sanqiang Zhao and Piyush Sharma and Tomer Levinboim and Radu Soricut},
  title = {{Informative Image Captioning with External Sources of Information}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=MY0_Zcskge},
}
Figure