VisualNews : Benchmark and Challenges in Entity-aware Image Captioning

Fuxiao Liu, Yinghan Wang, Tianlu Wang, Vicente Ordonez

Abstract

We propose Visual News Captioner, an entity-aware model for the task of news image captioning. We also introduce Visual News, a large-scale benchmark consisting of more than one million news images along with associated news articles, image captions, author information, and other metadata. Unlike the standard image captioning task, news images depict situations where people, locations, and events are of paramount importance. Our proposed method can effectively combine visual and textual features to generate captions with richer information such as events and entities. Built upon the Transformer architecture, our model is further equipped with novel multi-modal feature fusion techniques and attention mechanisms designed to generate named entities more accurately. Our method utilizes fewer parameters while achieving slightly better prediction results than competing methods. Our larger and more diverse Visual News dataset further highlights the remaining challenges in captioning news images.

Introduction: Beyond Generic Descriptions

Standard image captioning models have become proficient at describing everyday scenes, such as "a cat sitting on a mat." However, these descriptions are often generic and "interpretatively shallow." In journalism, a caption's value lies in its specificity: identifying who is in the image, where they are, and why the moment matters.

President Obama debating Mitt Romney: An example of an entity-aware news image requiring external context from an article.

The challenge is that visual information alone is insufficient to identify specific individuals or events. To caption a news image accurately, a model must "read" the associated news article to find relevant named entities (people, places, dates) and then "anchor" those entities to the correct parts of the image. Previous datasets were either too small or lacked the stylistic diversity of multiple global news agencies.

Methodology: The Visual News Captioner

The authors propose the Visual News Captioner, a Transformer-based architecture designed to be entity-aware. Key technical highlights include:

  • Visual Selective Layer: Instead of encoding the image and text in isolation, this layer uses a visual information gate to update textual embeddings, prioritizing visually relevant parts of the article.
  • Multi-head Pointer-Generator: To handle the "long tail" of rare names, the model uses a "pointer" mechanism to copy-paste named entities directly from the source article into the caption.
  • Tag-Cleaning Post-processing: The model uses placeholder tags (e.g., PERSON_) during training to solve the "Out-of-Vocabulary" (OOV) problem. During inference, it replaces these tags with the most frequent and relevant entities found in the article.
  • Attention on Attention (AoA): The model employs AoA modules to better filter and weigh the importance of different multimodal features, such as image patches versus article words.

The Visual News Benchmark

The paper introduces Visual News, currently the largest and most diverse dataset in the field, featuring over 1 million images and 600,000+ articles. Data is aggregated from four major agencies: The Guardian, BBC, USA TODAY, and The Washington Post. This dataset dwarfs previous benchmarks like GoodNews and NYTimes800k in both volume and source diversity. The model was tested across all three datasets to prove its generalizability.

Results and Insights

The Visual News Captioner outperformed all existing baselines across multiple metrics. On the new Visual News dataset, the model achieved a CIDEr score of 50.5, a significant leap over the 13.2 achieved by previous baselines. It also set new records on the GoodNews (55.4 CIDEr) and NYTimes800k (56.1 CIDEr) benchmarks.

News images often contain specific historical figures and complex contexts, requiring entity-aware models to distinguish between multiple subjects.

Key insights from the study include:

  • News Style Diversity: Different news agencies have distinct "personalities." For example, The Guardian tends to have longer captions, while USA TODAY uses a higher density of person entities.
  • The Agency Gap: A model trained on one news source performs significantly worse when tested on another, highlighting that news captioning involves learning specific editorial styles.
  • Efficiency: The Visual News Captioner achieves state-of-the-art results while using significantly fewer parameters (~93M) compared to previous models like "Transform and Tell" (~200M).

Conclusion and Future Work

While the "Tag-Cleaning" method helps with rare names, mapping text to pixels remains a fundamental difficulty. The "long tail" of entities is still a challenge; if a person is a minor figure in the article, the model may struggle to anchor them correctly. Future research could utilize this dataset for multimodal fake news detection or abstractive news summarization.

Summary updated

Abstract

We propose Visual News Captioner, an entity-aware model for the task of news image captioning. We also introduce Visual News, a large-scale benchmark consisting of more than one million news images along with associated news articles, image captions, author information, and other metadata. Unlike the standard image captioning task, news images depict situations where people, locations, and events are of paramount importance. Our proposed method can effectively combine visual and textual features to generate captions with richer information such as events and entities. More specifically, built upon the Transformer architecture, our model is further equipped with novel multi-modal feature fusion techniques and attention mechanisms, which are designed to generate named entities more accurately. Our method utilizes much fewer parameters while achieving slightly better prediction results than competing methods. Our larger and more diverse Visual News dataset further highlights the remaining challenges in captioning news images.
BibTeX
@inproceedings{liu2021visualnews,
  author = {Fuxiao Liu and Yinghan Wang and Tianlu Wang and Vicente Ordonez},
  title = {{VisualNews : Benchmark and Challenges in Entity-aware Image Captioning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=sR6CtZjDHrr},
}
Figure