G3raphGround: Graph-Based Language Grounding

Mohit Bajaj, Lanjun Wang, Leonid Sigal

Abstract

In this paper we present an end-to-end framework for grounding of phrases in images. In contrast to previous works, our model, which we call GraphGround, uses graphs to formulate more complex, non-sequential dependencies among proposal image regions and phrases. We capture intra-modal dependencies using a separate graph neural network for each modality (visual and lingual), and then use conditional message-passing in another graph neural network to fuse their outputs and capture cross-modal relationships. This final representation results in grounding decisions. The framework supports many-to-many matching and is able to ground single phrase to multiple image regions and vice versa. We validate our design choices through a series of ablation studies and illustrate state-of-the-art performance on Flickr30k and ReferIt Game benchmark datasets.
BibTeX
@inproceedings{bajaj2019g,
  author = {Mohit Bajaj and Lanjun Wang and Leonid Sigal},
  title = {{G3raphGround: Graph-Based Language Grounding}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  doi = {10.1109/iccv.2019.00438},
}

Loading related content...

Figure