VisDG: Towards Spontaneous Visual Dialogue Generation

Qichuan Yang, Liuxin Zhang, Yang Zhang, Jinghua GaoIJCNN 2021 · OpenReview

Abstract

Current vision and language understanding tasks can be generally divided into two categories: image (video) description and visual question answering. Both of them aim to train a model that either straightforwardly generates a description or answers the predefined questions based on an image or a sequence of images from a spectators perspective. However, a large proportion of real-world human interactions also involve spontaneous dialogue exchanges among multiple speakers as well as dynamic visual-textual context, which requires an AI agent to hold a natural and open-ended dialog with humans in a first-person manner based on both visual and textual context. To move closer towards achieving such a spontaneous multimodal conversation, we introduce a new visual dialogue generation dataset (VisDG) based on keyframes and corresponding subtitles extracted from Friends - an American sitcom television series. Specifically, given a start frame and its corresponding dialogue text, the agent has to generate both a meaningful textual response as well as a correct image candidate for the latter part of the dialogue turn. Furthermore, we also propose an end-to-end image-text synergistic network (ITSN) for the task, which outperforms several sophisticated baselines on the proposed VisDG.
BibTeX
@inproceedings{yang2021visdg,
  author = {Qichuan Yang and Liuxin Zhang and Yang Zhang and Jinghua Gao},
  title = {{VisDG: Towards Spontaneous Visual Dialogue Generation}},
  booktitle = {IJCNN},
  year = {2021},
  url = {https://openreview.net/forum?id=IWKHLfa18k},
}

Loading related content...

Figure