Look Deeper See Richer: Depth-aware Image Paragraph Captioning
Lists
Only you see this
Abstract
With the widespread availability of image captioning at a sentence level, how to automatically generate image paragraphs is yet well explored. Describing an image by a full paragraph involves organising sentences orderly, coherently and diversely, inevitably leading higher complexity than by a single sentence. Existing image paragraph captioning methods give a series of sentences to represent the objects and regions of interests, where the descriptions are essentially generated by feeding the image fragments containing objects and regions into conventional image single-sentence captioning models. This strategy is difficult to generate the descriptions that guarantee the stereoscopic hierarchy and non-overlapping objects. In this paper, we propose a Depth-aware Attention Model (\textitDAM ) to generate paragraph captions for images. The depths of image areas are firstly estimated in order to discriminate objects in a range of spatial locations, which can further guide the linguistic decoder to reveal spatial relationships among objects. This model completes the paragraph in a logical and coherent manner. By incorporating the attention mechanism, the learned model swiftly shifts the sentence focus during paragraph generation, whilst avoiding verbose descriptions on a same object. Extensive quantitative experiments and the user study have been conducted on the Visual Genome dataset, which demonstrate the effectiveness and the interpretability of the proposed model.
BibTeX
@inproceedings{wang2018look,
author = {Ziwei Wang and Yadan Luo and Yang Li and Zi Huang and Hongzhi Yin},
title = {{Look Deeper See Richer: Depth-aware Image Paragraph Captioning}},
booktitle = {ACM Multimedia},
year = {2018},
doi = {10.1145/3240508.3240583},
}Loading related content...