Visual-Semantic Graph Matching for Visual Grounding
Lists
Only you see this
Abstract
Visual Grounding is the task of associating entities in a natural language sentence with objects in an image. In this paper, we formulate visual grounding as a graph matching problem to find node correspondences between a visual scene graph and a language scene graph. These two graphs are heterogeneous, representing structure layouts of the sentence and image, respectively. We learn unified contextual node representations of the two graphs by using a cross-modal graph convolutional network to reduce their discrepancy. The graph matching is thus relaxed as a linear assignment problem because the learned node representations characterize both node information and structure information. A permutation loss and a semantic cycle-consistency loss are further introduced to solve the linear assignment problem with or without ground-truth correspondences. Experimental results on two visual grounding tasks, i.e., referring expression comprehension and phrase localization, demonstrate the effectiveness of our method.
BibTeX
@inproceedings{jing2020visual,
author = {Chenchen Jing and Yuwei Wu and Mingtao Pei and Yao Hu and Yunde Jia and Qi Wu},
title = {{Visual-Semantic Graph Matching for Visual Grounding}},
booktitle = {ACM Multimedia},
year = {2020},
doi = {10.1145/3394171.3413902},
}Loading related content...