Zero-Shot Grounding of Objects from Natural Language Queries

Arka Sadhu, Kan Chen, Ram NevatiaICCV 2019 · OpenReview

Abstract

Phrase grounding localizes specific objects in images based on natural language queries. Traditional systems often fail when encountering "unseen" nouns because they rely on fixed-category object detectors. We introduce ZSGNet, a single-stage model that bypasses this bottleneck by combining detection and grounding into a unified framework. By leveraging dense proposals and semantic word embeddings, ZSGNet enables Zero-Shot Grounding (ZSG) of novel objects. Our evaluation across four distinct difficulty conditions and new dataset splits demonstrates that ZSGNet achieves state-of-the-art performance on standard benchmarks and significantly outperforms baselines in zero-shot scenarios.

Introduction: The Detector Bottleneck

Phrase grounding is the task of mapping a textual description, such as "the red minivan behind the blue car," to a specific bounding box in an image. Most existing systems utilize a two-stage "detect-then-match" pipeline: first, an object detector (e.g., Faster R-CNN) generates potential object proposals; second, a grounding model selects the best match from these proposals.

Examples of phrase grounding across different categories and attributes, illustrating the variety of objects ZSGNet must handle.

The primary limitation of this approach is the detector's fixed vocabulary. If a detector is trained on a limited set of categories (like the 80 classes in COCO), it will fail to propose regions for novel objects like a "moped" or "beanie cap." This paper addresses this through Zero-Shot Grounding (ZSG), focusing on localizing objects described by nouns not seen during the training phase.

ZSGNet: A Single-Stage Architecture

To overcome the limitations of two-stage systems, we propose ZSGNet, a single-stage architecture that integrates proposal generation and grounding into a single end-to-end process.

Architecture Overview

  • Visual Backbone: ZSGNet uses a ResNet-50 backbone with a Feature Pyramid Network (FPN) to extract multi-scale visual features. This ensures the model can handle objects of varying sizes and resolutions.
  • Language Module: Queries are processed using a Bi-LSTM initialized with pre-trained GloVe embeddings. These embeddings are crucial for zero-shot performance as they provide semantic context; for instance, the model can relate an unseen "minivan" to a known "car" based on their proximity in the vector space.
  • Multi-Modal Fusion: Language features are fused with visual feature maps at every spatial location through tiling and concatenation, allowing the model to evaluate every part of the image against the query.
  • Dense Prediction Head: Instead of sparse proposals, ZSGNet evaluates a dense grid of anchor boxes. A Fully Convolutional Network (FCN) predicts a classification score (relevance to the query) and regression offsets (to refine the box) for each anchor.
  • Optimization: The model is trained using Focal Loss to address the extreme class imbalance between the single target object and the thousands of background anchors.

Evaluating Zero-Shot Grounding

Evaluating ZSG requires careful consideration of the relationship between seen and unseen categories. The authors define four conditions of increasing difficulty:

  1. Novel Words: Generalizing to synonyms or variations of seen categories.
  2. Novel Categories: Grounding objects from entirely new classes.
  3. Semantic Distance: Testing performance as the gap between seen and unseen concepts widens.
  4. Disambiguation: Distinguishing an unseen target from a similar "seen" distractor in the same image.

To support these evaluations, new benchmarks were created by sub-sampling Flickr30k Entities and Visual Genome to create specific splits (Flickr-Split-0/1 and VG-Split-2/3).

Results and Discussion

ZSGNet sets a new standard for phrase grounding performance:

  • Standard Benchmarks: On Flickr30k Entities, ZSGNet achieved 63.39% accuracy. On ReferIt, it reached 58.63%, a substantial improvement over previous two-stage models which hovered around 44%.
  • Zero-Shot Performance: In the "novel word" setting (Flickr-Split-0), ZSGNet achieved 43.02%, significantly outperforming the baseline (35.62%).
  • Semantic Robustness: Experiments on Visual Genome splits showed that ZSGNet's performance degrades gracefully as semantic distance increases, confirming its ability to leverage linguistic relationships rather than just memorizing patterns.

Conclusion

ZSGNet demonstrates that a single-stage, detector-free approach is superior for grounding novel objects. By treating grounding as a dense prediction task and utilizing semantic embeddings, ZSGNet effectively bridges the gap between vision and language for open-vocabulary scenarios. Future work will focus on improving complex spatial reasoning and handling linguistic polysemy.

Summary updated

Abstract

A phrase grounding system localizes a particular object in an image referred to by a natural language query. In previous work, the phrases were restricted to have nouns that were encountered in training, we extend the task to Zero-Shot Grounding(ZSG) which can include novel, "unseen" nouns. Current phrase grounding systems use an explicit object detection network in a 2-stage framework where one stage generates sparse proposals and the other stage evaluates them. In the ZSG setting, generating appropriate proposals itself becomes an obstacle as the proposal generator is trained on the entities common in the detection and grounding datasets. We propose a new single-stage model called ZSGNet which combines the detector network and the grounding system and predicts classification scores and regression parameters. Evaluation of ZSG system brings additional subtleties due to the influence of the relationship between the query and learned categories; we define four distinct conditions that incorporate different levels of difficulty. We also introduce new datasets, sub-sampled from Flickr30k Entities and Visual Genome, that enable evaluations for the four conditions. Our experiments show that ZSGNet achieves state-of-the-art performance on Flickr30k and ReferIt under the usual "seen" settings and performs significantly better than baseline in the zero-shot setting.
BibTeX
@inproceedings{sadhu2019zero,
  author = {Arka Sadhu and Kan Chen and Ram Nevatia},
  title = {{Zero-Shot Grounding of Objects from Natural Language Queries}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2019},
  url = {https://openreview.net/forum?id=igvfXjEK_aO},
}
Figure