Improving Image Paragraph Captioning with Dual Relations
Lists
Only you see this
Abstract
Image paragraph captioning aims to generate multiple de-scriptive sentences for an image. However, most previous methods ignore the explicit relations among objects resulting in unsatisfactory performance. In this paper, we propose a novel model (i.e., DualRel) to capture spatial and seman-tic relations among objects. Specifically, the spatial relation embedding is obtained solely from images using a predefined geometry pattern. With the help of captions, the semantic relation embedding is learned in a weakly supervised man-ner. These two relation embeddings are then interacted with regional features of objects through a relation-aware attention interaction. It first obtains a visual context vector using regional features. Then with the visual context vector, we obtain the corresponding spatial and semantic relation-aware vectors using attentions. These three vectors are fused with two gates for language decoding to further generate a para-graph. Experimental results on Stanford benchmark dataset show that DualRel achieves remarkable improvements <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> Code released at https://github.com/fuyunll07/DualRel.
BibTeX
@inproceedings{liu2022improving,
author = {Yun Liu and Yihui Shi and Fangxiang Feng and Ruifan Li and Zhanyu Ma and Xiaojie Wang},
title = {{Improving Image Paragraph Captioning with Dual Relations}},
booktitle = {ICME},
year = {2022},
doi = {10.1109/icme52920.2022.9859701},
}Loading related content...