Semantically Guided Visual Question Answering

Handong Zhao, Quanfu Fan, Dan Gutfreund, Yun Fu

Abstract

We present a novel approach to enhance the challenging task of Visual Question Answering (VQA) by incorporating and enriching semantic knowledge in a VQA model. We first apply Multiple Instance Learning (MIL) to extract a richer visual representation addressing concepts beyond objects such as actions and colors. Motivated by the observation that semantically related answers often appear together in prediction, we further develop a new semantically-guided loss function for model learning which has the potential to drive weakly-scored but correct answers to the top while suppressing wrong answers. We show that these two ideas contribute to performance improvement in a complementary way. We demonstrate competitive results comparable to the state of the art on two VQA benchmark datasets.
BibTeX
@inproceedings{zhao2018semantically,
  author = {Handong Zhao and Quanfu Fan and Dan Gutfreund and Yun Fu},
  title = {{Semantically Guided Visual Question Answering}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2018},
  doi = {10.1109/wacv.2018.00205},
}

Loading related content...

Figure