Visual Question Answering With Dense Inter- and Intra-Modality Interactions
Lists
Only you see this
Abstract
Learning effective interactions between multi-modal features is at the heart of visual question answering (VQA). A common defect of the existing VQA approaches is that they only consider a very limited amount of inter-modality interactions, which may be not enough to model latent complex image-question relations that are necessary for accurately answering questions. Besides, most methods neglect the modeling of the intra-modality interactions that is also important to VQA. In this work, we propose a novel <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DenIII</i> framework for modeling dense inter-, and intra-modality interactions. It densely connects all pairwise layers of the network via the proposed Inter-, and Intra-modality Attention Connectors, capturing fine-grained interplay across all hierarchical levels. The Inter-modality Attention Connector efficiently connects the multi-modality features at any two layers with bidirectional attention, capturing the inter-modality interactions. While the Intra-modality Attention Connector connects the features of the same modality with unidirectional attention, and models the intra-modality interactions. Extensive ablation studies, and visualizations validate the effectiveness of our method, and <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DenIII</i> achieves state-of-the-art or competitive performance on three publicly available datasets.
BibTeX
@article{liu2021visual,
author = {Fei Liu and Jing Liu and Zhiwei Fang and Richang Hong and Hanqing Lu},
title = {{Visual Question Answering With Dense Inter- and Intra-Modality Interactions}},
journal = {IEEE Trans. Multimedia},
year = {2021},
url = {https://openreview.net/forum?id=eyyOl2W9Ey},
}Loading related content...