Knowledge Prompt Makes Composed Pre-Trained Models Zero-Shot News Captioner

Yanhui Wang, Ning Xu, Hongshuo Tian, Bo Lv, Yulong Duan, Xuanya Li, An-An Liu

Abstract

News image captioning aims to generate descriptions containing concrete named entities for news images by leveraging relevant news articles. However, existing approaches suffer from two shortcomings: 1) lack of commonsense knowledge required to understand named entities, and 2) limited multimodal context modeling capabilities. In this paper, we propose to migrate the ability of large-scale pre-trained models for news image captioning. To acquire factual knowledge for describing named entities, we induce a pre-trained language model for commonsense knowledge reasoning using context-aware knowledge prompts. To compose a new multimodal context modeling capability, we coordinate pre-trained models by a unified language representation and constrain joint multimodal context reasoning with cross-modal consistency objective. Experimental results on GoodNews and NYTimes datasets show that our proposed method exhibits considerable captioning capabilities even without training on news data.
BibTeX
@inproceedings{wang2023knowledge,
  author = {Yanhui Wang and Ning Xu and Hongshuo Tian and Bo Lv and Yulong Duan and Xuanya Li and An-An Liu},
  title = {{Knowledge Prompt Makes Composed Pre-Trained Models Zero-Shot News Captioner}},
  booktitle = {ICME},
  year = {2023},
  doi = {10.1109/icme55011.2023.00489},
}

Loading related content...

Figure