Abstract
Phrase grounding localizes specific objects in images based on natural language queries. Traditional systems often fail when encountering "unseen" nouns because they rely on fixed-category object detectors. We introduce ZSGNet, a single-stage model that bypasses this bottleneck by combining detection and grounding into a unified framework. By leveraging dense proposals and semantic word embeddings, ZSGNet enables Zero-Shot Grounding (ZSG) of novel objects. Our evaluation across four distinct difficulty conditions and new dataset splits demonstrates that ZSGNet achieves state-of-the-art performance on standard benchmarks and significantly outperforms baselines in zero-shot scenarios.
Introduction: The Detector Bottleneck
Phrase grounding is the task of mapping a textual description, such as "the red minivan behind the blue car," to a specific bounding box in an image. Most existing systems utilize a two-stage "detect-then-match" pipeline: first, an object detector (e.g., Faster R-CNN) generates potential object proposals; second, a grounding model selects the best match from these proposals.
The primary limitation of this approach is the detector's fixed vocabulary. If a detector is trained on a limited set of categories (like the 80 classes in COCO), it will fail to propose regions for novel objects like a "moped" or "beanie cap." This paper addresses this through Zero-Shot Grounding (ZSG), focusing on localizing objects described by nouns not seen during the training phase.
ZSGNet: A Single-Stage Architecture
To overcome the limitations of two-stage systems, we propose ZSGNet, a single-stage architecture that integrates proposal generation and grounding into a single end-to-end process.
Architecture Overview
- Visual Backbone: ZSGNet uses a ResNet-50 backbone with a Feature Pyramid Network (FPN) to extract multi-scale visual features. This ensures the model can handle objects of varying sizes and resolutions.
- Language Module: Queries are processed using a Bi-LSTM initialized with pre-trained GloVe embeddings. These embeddings are crucial for zero-shot performance as they provide semantic context; for instance, the model can relate an unseen "minivan" to a known "car" based on their proximity in the vector space.
- Multi-Modal Fusion: Language features are fused with visual feature maps at every spatial location through tiling and concatenation, allowing the model to evaluate every part of the image against the query.
- Dense Prediction Head: Instead of sparse proposals, ZSGNet evaluates a dense grid of anchor boxes. A Fully Convolutional Network (FCN) predicts a classification score (relevance to the query) and regression offsets (to refine the box) for each anchor.
- Optimization: The model is trained using Focal Loss to address the extreme class imbalance between the single target object and the thousands of background anchors.
Evaluating Zero-Shot Grounding
Evaluating ZSG requires careful consideration of the relationship between seen and unseen categories. The authors define four conditions of increasing difficulty:
- Novel Words: Generalizing to synonyms or variations of seen categories.
- Novel Categories: Grounding objects from entirely new classes.
- Semantic Distance: Testing performance as the gap between seen and unseen concepts widens.
- Disambiguation: Distinguishing an unseen target from a similar "seen" distractor in the same image.
To support these evaluations, new benchmarks were created by sub-sampling Flickr30k Entities and Visual Genome to create specific splits (Flickr-Split-0/1 and VG-Split-2/3).
Results and Discussion
ZSGNet sets a new standard for phrase grounding performance:
- Standard Benchmarks: On Flickr30k Entities, ZSGNet achieved 63.39% accuracy. On ReferIt, it reached 58.63%, a substantial improvement over previous two-stage models which hovered around 44%.
- Zero-Shot Performance: In the "novel word" setting (Flickr-Split-0), ZSGNet achieved 43.02%, significantly outperforming the baseline (35.62%).
- Semantic Robustness: Experiments on Visual Genome splits showed that ZSGNet's performance degrades gracefully as semantic distance increases, confirming its ability to leverage linguistic relationships rather than just memorizing patterns.
Conclusion
ZSGNet demonstrates that a single-stage, detector-free approach is superior for grounding novel objects. By treating grounding as a dense prediction task and utilizing semantic embeddings, ZSGNet effectively bridges the gap between vision and language for open-vocabulary scenarios. Future work will focus on improving complex spatial reasoning and handling linguistic polysemy.
Summary updated