Neural Sequential Phrase Grounding (SeqGROUND)

Pelin Dogan, Leonid Sigal, Markus H. Gross

Abstract

We propose an end-to-end approach for phrase grounding in images. Unlike prior methods that typically attempt to ground each phrase independently by building an image-text embedding, our architecture formulates grounding of multiple phrases as a sequential and contextual process. Specifically, we encode region proposals and all phrases into two stacks of LSTM cells, along with so-far grounded phrase-region pairs. These LSTM stacks collectively capture context for grounding of the next phrase. The resulting architecture, which we call SeqGROUND, supports many-to-many matching by allowing an image region to be matched to multiple phrases and vice versa. We show competitive performance on the Flickr30K benchmark dataset and, through ablation studies, validate the efficacy of sequential grounding as well as individual design choices in our model architecture.

Introduction

Phrase grounding—the task of localizing natural language phrases within an image using bounding boxes—is a cornerstone of vision-language understanding. While object detection has matured, phrase grounding remains challenging because it requires deciphering complex relationships between entities and their attributes.

Traditional models often treat phrases in a sentence as independent entities, matching them to image regions in isolation. This approach ignores critical contextual cues. For instance, identifying a specific "lady" in a crowded scene might depend on first locating the "bouquet" she is holding. SeqGROUND addresses this by reframing phrase grounding as a sequential and contextual decision-making process where previous grounding decisions inform subsequent ones.

The SeqGROUND Architecture

The core of SeqGROUND is an end-to-end trainable architecture that manages information through three distinct Recurrent Neural Network (LSTM) stacks.

The SeqGROUND architecture featuring Phrase, Box, and History stacks for contextual grounding.
  1. The Phrase Stack: Encodes the sequence of phrases. Research indicates that processing phrases in reverse lexical order (right-to-left) is most effective, as trailing modifiers often provide the necessary context to disambiguate primary subjects.
  2. The Box Stack: A bi-directional LSTM that encodes region proposals (e.g., from Faster R-CNN) ordered spatially to capture the scene's layout.
  3. The History Stack: Tracks the state of the grounding process by storing features of previously matched phrase-region pairs.

At each step, the model integrates the current phrase, available boxes, global image context, and the history stack to determine the best match. This design naturally supports many-to-many matching, allowing a single phrase to link to multiple boxes (e.g., "a group of people") and vice versa.

Experimental Results

The model was evaluated on the Flickr30K Entities dataset, which consists of 31,783 images with five descriptive sentences each.

Ablation study results and accuracy comparison across different phrase orders and directions.

SeqGROUND achieved a state-of-the-art accuracy of 61.60%, surpassing previous benchmarks. Ablation studies (summarized in the table above) highlight the importance of each component. Removing the History Stack (NH) reduced accuracy to 59.55%, while removing visual context (NI) dropped it to 60.34%. Furthermore, the model demonstrates superior performance in handling complex "many-to-many" scenarios compared to baseline methods.

Discussion and Future Work

While SeqGROUND advances the state-of-the-art, it remains dependent on the quality of initial region proposals from the RPN. Small or obscure objects missed by the detector cannot be grounded. Additionally, the sequential nature introduces higher computational complexity compared to parallel matching models. Future research may explore Reinforcement Learning to optimize the grounding "policy" or integrate more advanced object detection backbones to mitigate the RPN bottleneck.

Summary updated

Abstract

We propose an end-to-end approach for phrase grounding in images. Unlike prior methods that typically attempt to ground each phrase independently by building an image-text embedding, our architecture formulates grounding of multiple phrases as a sequential and contextual process. Specifically, we encode region proposals and all phrases into two stacks of LSTM cells, along with so-far grounded phrase-region pairs. These LSTM stacks collectively capture context for grounding of the next phrase. The resulting architecture, which we call SeqGROUND, supports many-to-many matching by allowing an image region to be matched to multiple phrases and vice versa. We show competitive performance on the Flickr30K benchmark dataset and, through ablation studies, validate the efficacy of sequential grounding as well as individual design choices in our model architecture.
BibTeX
@inproceedings{dogan2019neural,
  author = {Pelin Dogan and Leonid Sigal and Markus H. Gross},
  title = {{Neural Sequential Phrase Grounding (SeqGROUND)}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2019},
  url = {https://openreview.net/forum?id=fli7_4MomSL},
}
Figure