Knowledge Prompt Makes Composed Pre-Trained Models Zero-Shot News Captioner
Lists
Only you see this
Abstract
News image captioning aims to generate descriptions containing concrete named entities for news images by leveraging relevant news articles. However, existing approaches suffer from two shortcomings: 1) lack of commonsense knowledge required to understand named entities, and 2) limited multimodal context modeling capabilities. In this paper, we propose to migrate the ability of large-scale pre-trained models for news image captioning. To acquire factual knowledge for describing named entities, we induce a pre-trained language model for commonsense knowledge reasoning using context-aware knowledge prompts. To compose a new multimodal context modeling capability, we coordinate pre-trained models by a unified language representation and constrain joint multimodal context reasoning with cross-modal consistency objective. Experimental results on GoodNews and NYTimes datasets show that our proposed method exhibits considerable captioning capabilities even without training on news data.
BibTeX
@inproceedings{wang2023knowledge,
author = {Yanhui Wang and Ning Xu and Hongshuo Tian and Bo Lv and Yulong Duan and Xuanya Li and An-An Liu},
title = {{Knowledge Prompt Makes Composed Pre-Trained Models Zero-Shot News Captioner}},
booktitle = {ICME},
year = {2023},
doi = {10.1109/icme55011.2023.00489},
}Loading related content...