An Attention-based Regression Model for Grounding Textual Phrases in Images

Ko Endo, Masaki Aono, Eric Nichols, Kotaro FunakoshiIJCAI 2017 · DOI

TL;DR

This paper introduces a novel attention-based regression model for grounding textual phrases in images, directly predicting bounding boxes without relying on external region proposal methods. By combining image and text attention mechanisms, the model achieves state-of-the-art accuracy on the ReferIt dataset, demonstrating a significant improvement in performance and efficiency.

What Problem Does This Paper Address?

The task of "grounding" involves identifying the specific region in an image that a given textual phrase refers to. This is a fundamental challenge in visual language understanding with applications ranging from image search to robotics. Previous approaches often treated this as a ranking problem, relying on pre-generated candidate regions. This method has two main drawbacks: performance is limited by the quality of the region proposal algorithm, and it incurs additional computational cost. Furthermore, if the correct region isn't perfectly captured by a candidate proposal, these methods can struggle. This paper aims to overcome these limitations by directly regressing the bounding box coordinates of the target region.

Methods

The proposed method employs a deep learning architecture with several key components:

  • Image Representation: A Convolutional Neural Network (CNN), specifically VGG 16, is used to extract image features from the conv5-3 layer. These features represent localized regions within the image.
  • Text Representation: Bidirectional Long Short-Term Memory networks (BLSTMs) are used to process the textual phrase, capturing word dependencies and generating word-level and phrase-level representations. BLSTMs are chosen to avoid bias towards words at the end of the phrase.
  • Image Attention Model: This component learns to estimate the relevance of each image region to the textual phrase. It uses a Multi-Layer Perceptron (MLP) to compute attention weights for each image region based on its features and the overall phrase representation.
  • Text Attention Model: This model refines the phrase representation by determining the importance of each word in the textual phrase, guided by the image attention map. This helps the model focus on the most relevant words for grounding.
  • Region Prediction Model: Finally, an MLP takes the attended image and text representations as input and directly predicts the bounding box of the target region in terms of its top-left corner coordinates (x, y) and its width and height (w, h). This is framed as a regression problem.

A novel aspect is the direct regression of bounding box coordinates, eliminating the need for candidate region proposals. The authors also introduce an image attention area-sensitive loss function to encourage the attention model to focus more precisely on the ground truth region.

Datasets

The primary dataset used for training and evaluation is the ReferIt dataset. This dataset comprises images, annotated regions within those images, and corresponding textual captions for each region. The authors note that ReferIt presents a data sparsity challenge compared to larger datasets like MSCOCO and VisualQA. They use the same data splits as previous work for fair comparison.

Key Insights and Ideas

  • Regression for Grounding: Treating grounding as a direct regression problem, rather than a ranking problem over proposals, is a significant departure that simplifies the pipeline and improves performance.
  • Synergistic Attention: Combining both image and text attention mechanisms allows the model to learn fine-grained correspondences between visual regions and textual elements, leading to more accurate grounding.
  • Image Attention Focus: The image attention model is crucial for localizing the target region, and its effectiveness is highlighted by the error analysis, where incorrect image attention is the most common error cause.
  • Loss Penalty for Precision: The introduction of an image attention area-sensitive loss function helps to refine the attention mechanism, leading to more precise localization and improved accuracy.

Key Results

The proposed attention-based regression model achieves a new state-of-the-art accuracy of 37.26% on the ReferIt dataset, surpassing the previous best by over 5 percentage points. Ablation studies show that:

  • Combining text and image attention improves performance by 7.83% over a model without either.
  • The image attention component contributes significantly to this improvement.
  • The proposed regression approach outperforms previous methods that score candidate regions.
  • The image attention area-sensitive loss penalty provides a further improvement of 0.85%. The paper also compares different word embeddings, finding that GloVe embeddings trained on a larger dataset yield better results.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Error Analysis: The most common error cause is incorrect image attention, often appearing too spread out. Text attention can still be biased towards the end of a sentence.
  • Ambiguous Annotations: The ReferIt dataset contains ambiguous ground truth regions and textual phrases, contributing to errors.
  • Hierarchical/Cyclical Attention: To address issues with attention bias and potentially improve handling of complex expressions, the authors suggest exploring hierarchical or cyclical attention models that allow for multiple passes over attention modules.
  • Objectness Judgement: Despite avoiding external region proposals, the model could potentially benefit from incorporating some form of "objectness" judgment into its region prediction.
  • Refining Penalty Function: Future work will focus on refining the penalty function, potentially incorporating measures of objectness.

Summary updated

BibTeX
@inproceedings{endo2017attention,
  author = {Ko Endo and Masaki Aono and Eric Nichols and Kotaro Funakoshi},
  title = {{An Attention-based Regression Model for Grounding Textual Phrases in Images}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2017},
  doi = {10.24963/ijcai.2017/558},
}
Figure