Informative Image Captioning with External Sources of Information

Sanqiang Zhao, Piyush Sharma, Tomer Levinboim, Radu Soricut

Introduction

Standard image captioning models often suffer from a "generality" problem. When presented with an image of a famous musician on stage, a typical model might generate a safe but bland description like "a person playing a guitar." While technically correct, this lacks the specific detail—such as the person's identity or the event name—that makes a caption truly useful.

Researchers from Google and the University of Pittsburgh have addressed this "informativeness gap" by developing a system that moves beyond generic hypernyms (broad category names like "person" or "building") to highly specific entities. By integrating external sources of information, their model can generate captions as precise as "Eric Clapton at the Crossroads Guitar Festival."

Informative captioning example: moving from generic descriptions to specific entities like Eric Clapton.

The Informativeness Gap

The tendency for models to default to generic language stems from two primary challenges:

  1. Task Complexity: Learning the broad structure of natural language while simultaneously mastering the fine-grained recognition of millions of unique global entities is a monumental task for a single model.
  2. Data Limitations: Large-scale training datasets often use simplified captions to ensure learning stability, which inadvertently trains models to avoid specific details.

To bridge this gap, the authors propose a model that can "look up" specific information from external tools—such as object detectors or web entity recognizers—and seamlessly weave those details into fluent, natural-sounding sentences.

Methods: A Multi-Encoder Architecture

The core of the proposed system is a multi-modal, multi-encoder Transformer. Unlike standard models that rely solely on visual pixels, this architecture utilizes three distinct "eyes" to understand an image:

  • Image Encoder: A standard CNN that extracts visual features from raw pixels.
  • Object Label Encoder: Processes a list of common objects detected in the scene (e.g., "guitar," "stage").
  • Web Entity Encoder: Processes fine-grained entities identified via external APIs, such as the Google Cloud Vision API (e.g., "Eric Clapton," "Crossroads Guitar Festival").
The multi-encoder Transformer architecture featuring gated attention for image, object, and web entity inputs.

Key Innovations

To manage these diverse inputs, the researchers introduced several technical innovations:

  • Multi-gated Decoder: A modified Transformer decoder uses "soft switches" (gates) to adaptively weight information from the three encoders at every step of word generation. This allows the model to decide, for instance, when to rely on the visual image for structure and when to pull a specific name from the web entity encoder.
  • Label Coverage Control: The model includes "control knobs" ($W_{obj}$ and $W_{we}$) that allow users to tune the level of detail at inference time. By "turning up the volume" on entity labels, the system can be forced to produce more detailed, albeit potentially riskier, captions.
  • Selective Hypernym Substitution: During training, the authors ensured ground-truth captions only contained specific names if those names were present in the input labels. If a name was missing from the input, it was swapped for a generic category to prevent the model from learning to "hallucinate" details it couldn't actually see.

Datasets and Training

The study utilized the Conceptual Captions dataset, containing 3.3 million image-caption pairs. The authors "re-processed" this data, mapping generic terms back to their original, specific entities using source Alt-text to create a more challenging and informative training set. External labels were generated using the Google Cloud Vision API to simulate real-world deployment where a model would rely on third-party recognition tools.

Key Results and Insights

The model's performance was evaluated both quantitatively and through human judgment:

  • Human Evaluation: In side-by-side comparisons, the proposed model was judged 24.33% more informative and 7.79% more correct than previous state-of-the-art baselines, with no significant loss in linguistic fluency.
  • Quantitative Success: The model achieved a CIDEr score of 87.25 when using specific names, a massive improvement over image-only baselines that typically score between 51 and 62.
  • Qualitative Precision: The model demonstrated a remarkable ability to identify specific locations (e.g., "Tromsø"), specific food types (e.g., "Basmati rice"), and even specific car models (e.g., "Kia Ceed") where generic models would only output "city," "food," or "car."

Conclusion and Future Work

While the model represents a significant step forward, it remains dependent on the quality of its "upstream" labelers; if an external API misidentifies an object, the captioning model is likely to propagate that error. Future research may focus on integrating Knowledge Graphs for deeper context or employing reinforcement learning to further refine the balance between informativeness and fluency. Ultimately, this work demonstrates that by treating images as the primary source and external labels as essential "hints," we can create AI that describes the world with the specificity and nuance of a human observer.

Summary updated

Abstract

An image caption should fluently present the essential information in a given image, including informative, fine-grained entity mentions and the manner in which these entities interact. However, current captioning models are usually trained to generate captions that only contain common object names, thus falling short on an important "informativeness" dimension. We present a mechanism for integrating image information together with fine-grained labels (assumed to be generated by some upstream models) into a caption that describes the image in a fluent and informative manner. We introduce a multimodal, multi-encoder model based on Transformer that ingests both image features and multiple sources of entity labels. We demonstrate that we can learn to control the appearance of these entity labels in the output, resulting in captions that are both fluent and informative.
BibTeX
@inproceedings{zhao2019informative,
  author = {Sanqiang Zhao and Piyush Sharma and Tomer Levinboim and Radu Soricut},
  title = {{Informative Image Captioning with External Sources of Information}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=MY0_Zcskge},
}

Loading related content...

Figure