Improving Zero-Shot Phrase Grounding via Reasoning on External Knowledge and Spatial Relations

Zhan Shi, Yilin Shen, Hongxia Jin, Xiaodan ZhuAAAI 2022 · OpenReview

Abstract

Phrase grounding is a multi-modal problem that localizes a particular noun phrase in an image referred to by a text query. In the challenging zero-shot phrase grounding setting, the existing state-of-the-art grounding models have limited capacity in handling the unseen phrases. Humans, however, can ground novel types of objects in images with little effort, significantly benefiting from reasoning with commonsense. In this paper, we design a novel phrase grounding architecture that builds multi-modal knowledge graphs using external knowledge and then performs graph reasoning and spatial relation reasoning to localize the referred nouns phrases. We perform extensive experiments on different zero-shot grounding splits sub-sampled from the Flickr30K Entity and Visual Genome dataset, demonstrating that the proposed framework is orthogonal to backbone image encoders and outperforms the baselines by 2~3% in accuracy, resulting in a significant improvement under the standard evaluation metrics.

Introduction

Phrase grounding is a fundamental task in multi-modal AI, requiring the system to map a specific noun phrase (e.g., "a walking stick") to a corresponding bounding box within an image. While current state-of-the-art models excel at localizing familiar objects, they often fail in zero-shot scenarios—where the query involves novel phrases or object categories not encountered during training.

The primary hurdle is the lack of "object priors" for unseen entities. If a model has never seen a "walking stick," it lacks the visual features necessary for direct detection. Humans overcome this by leveraging reasoning: if we identify a person and know that "people hold sticks," we can infer the stick's location by searching near the person's hand. This research aims to replicate this human-like reasoning by integrating external knowledge and spatial relationship analysis.

Motivation for zero-shot phrase grounding using external knowledge and reasoning.

Methodology

The proposed framework employs a three-stage architecture designed to facilitate reasoning over novel entities:

1. Multi-Modal Knowledge Graph (MMKG) Construction

For every image-query pair, the model constructs a graph consisting of:

  • Entity Nodes: Derived from the text query.
  • Object Nodes: Identified using an off-the-shelf detector (e.g., Detectron2) to serve as "anchor" objects (like "people").
  • Relation Nodes: These connect entities and objects using external knowledge from WordNet and ConceptNet (e.g., IsA, PartOf) and semantic relations from Visual Genome scene graphs (e.g., man-wearing-hat).

2. Graph Reasoning

The model utilizes Graph Convolutional Networks (GCNs) to propagate information across the MMKG. This process allows the representation of a novel entity (e.g., "walking stick") to be informed by the visual features and spatial locations of related, known anchor objects ("people").

3. Spatial Relation Reasoning (SRR)

To refine localization, the model predicts eight types of spatial relations (e.g., Inside, Cover, Overlap, Top-Left) between candidate proposals and detected anchor objects. The SRR module acts as a geometric constraint, narrowing the search space based on the expected relative position of the target object to a known landmark.

Experimental Evaluation

The model was evaluated on several challenging sub-sampled splits from the Flickr30K Entity and Visual Genome (VG) datasets, specifically designed to test zero-shot capabilities (e.g., Flickr-0 for novel phrases and Flickr-1 for novel categories).

Key Results

  • Performance Gains: The framework consistently outperformed baselines (such as ZSG and QRG) by 2–3% in accuracy across zero-shot splits, achieving 45.99% on Flickr-0 and 33.12% on Flickr-1.
  • Architecture Agnostic: Improvements were consistent across different backbones (VGG16 and ResNet-50), demonstrating that the reasoning framework is complementary to advancements in visual encoders.
  • Ablation Studies: Removing relation nodes or the SRR module led to significant performance drops, confirming that both external knowledge and geometric reasoning are essential for zero-shot success.

Conclusion

By bridging the gap between visual detection and symbolic reasoning, this work demonstrates that external knowledge can serve as a powerful prior for grounding unseen objects. While challenges remain regarding the granularity of knowledge and linguistic nuances, the integration of multi-modal knowledge graphs and spatial reasoning marks a significant step toward more robust and human-like visual understanding.

Summary updated

Abstract

Phrase grounding is a multi-modal problem that localizes a particular noun phrase in an image referred to by a text query. In the challenging zero-shot phrase grounding setting, the existing state-of-the-art grounding models have limited capacity in handling the unseen phrases. Humans, however, can ground novel types of objects in images with little effort, significantly benefiting from reasoning with commonsense. In this paper, we design a novel phrase grounding architecture that builds multi-modal knowledge graphs using external knowledge and then performs graph reasoning and spatial relation reasoning to localize the referred nouns phrases. We perform extensive experiments on different zero-shot grounding splits sub-sampled from the Flickr30K Entity and Visual Genome dataset, demonstrating that the proposed framework is orthogonal to backbone image encoders and outperforms the baselines by 2~3% in accuracy, resulting in a significant improvement under the standard evaluation metrics.
BibTeX
@inproceedings{shi2022improving,
  author = {Zhan Shi and Yilin Shen and Hongxia Jin and Xiaodan Zhu},
  title = {{Improving Zero-Shot Phrase Grounding via Reasoning on External Knowledge and Spatial Relations}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=WMrxfTiS8p},
}
Figure