Interpretable and Globally Optimal Prediction for Textual Grounding using Image Concepts

Raymond A. Yeh, Jinjun Xiong, Wen-mei W. Hwu, Minh N. Do, Alexander G. Schwing

Introduction

Textual grounding is a critical task in computer vision and natural language processing that involves linking specific text phrases to their corresponding spatial regions in an image. For instance, in a robotic application, grounding the command "pick up the bottle on the top shelf" requires the system to identify the precise coordinates of that specific bottle. Traditionally, this has been approached as a two-step process: generating a set of "region proposals" and then ranking them. However, this method is limited by the quality of the initial guesses; if the proposal generator fails to include the correct object, the system cannot succeed.

This paper introduces a novel framework that bypasses the proposal bottleneck by treating grounding as a global energy minimization problem. By searching over all possible bounding boxes using an efficient branch-and-bound algorithm, the model ensures a globally optimal prediction while maintaining human-interpretable word embeddings.

Example of textual grounding where multiple entities (person, ball, lanes) are localized within a complex scene.

Methodology: Image Concepts and Score Maps

The researchers propose a unified framework that utilizes "image concepts" rather than raw pixels. These concepts are pre-computed score maps derived from three primary sources:

  1. Semantic Segmentation: Identifying regions like "grass" or "sky."
  2. Object Detection: Locating specific objects like "cars" or "people."
  3. Spatial Priors: Incorporating geometric cues such as "left," "right," or "center."

The model learns a linear weight vector ($w$) that connects words to these image concepts. For example, the system learns that the word "ship" should highly correlate with the "boat" image concept. This linearity ensures that the model's decision-making process is transparent and interpretable.

Global Optimization via Branch and Bound

To avoid the limitations of pre-generated proposals, the authors employ a Branch and Bound algorithm. This technique mathematically prunes the search space, allowing the system to efficiently evaluate millions of potential bounding boxes. It identifies the box that maximizes the accumulated score of the relevant image concepts. To train the parameters, the authors use a Structured Support Vector Machine (SVM) approach, which penalizes the model more heavily if the predicted box has a low Intersection over Union (IoU) with the ground truth.

Experimental Evaluation

The framework was evaluated on two major benchmarks: Flickr 30k Entities (31k images with 275k annotated noun phrases) and ReferItGame (20k images with 99k regions). The latter is particularly challenging as it involves referential language, such as "the man on the right."

Grounding a specific person in a referential context, demonstrating the model's ability to distinguish between similar entities.

Key Results

At the time of publication, the model established new state-of-the-art (SOTA) performance standards:

  • Accuracy: Outperformed existing methods by 3.08% on Flickr 30k and 7.77% on ReferItGame.
  • Efficiency: Despite the exhaustive search, the branch-and-bound implementation was highly efficient, with the scoring phase taking approximately 1.05ms on a CPU.
  • Robustness: The model showed superior performance across diverse phrase types, including people, animals, and vehicles.

Discussion and Future Work

A significant insight from this research is that spatial priors are incredibly powerful; knowing that "floor" is typically at the bottom of an image provides a strong baseline even without complex visual analysis. However, the model's reliance on linear combinations means it may struggle with compound concepts like "dirt bike" if it simply sums the scores for "dirt" and "bike."

Future work may involve exploring non-linear energy functions and integrating the branch-and-bound search directly into deep neural network architectures for end-to-end training, potentially allowing for the fine-tuning of visual features specifically for the grounding task.

Summary updated

BibTeX
@inproceedings{yeh2017interpretable,
  author = {Raymond A. Yeh and Jinjun Xiong and Wen-mei W. Hwu and Minh N. Do and Alexander G. Schwing},
  title = {{Interpretable and Globally Optimal Prediction for Textual Grounding using Image Concepts}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2017},
  url = {https://openreview.net/forum?id=HkZaeDWObH},
}
Figure