Proposal-free One-stage Referring Expression via Grid-Word Cross-Attention

Wei Suo

Abstract

Referring Expression Comprehension (REC) is a fundamental task in computer vision and natural language processing that involves localizing a target object in an image based on a linguistic description. Traditional one-stage methods often rely on pre-defined anchors, which can lead to modality misalignment and reduced generalization across datasets. This paper presents PFOS (Proposal-Free One-Stage), a framework that eliminates anchors in favor of a bidirectional grid-word cross-attention mechanism. By directly regressing object coordinates from multi-modal feature maps, PFOS achieves state-of-the-art accuracy while significantly outperforming existing methods in inference speed.

Introduction

The goal of REC is to bridge the gap between visual perception and natural language understanding. Historically, the field has been dominated by two-stage methods that generate region proposals and then rank them, or one-stage methods that utilize anchors. However, two-stage approaches are computationally intensive, and anchor-based one-stage models often struggle with the fine-grained details of complex queries. The proposed PFOS model addresses these challenges by removing the anchor dependency and implementing a more effective modality fusion strategy.

Methodology

The PFOS architecture is designed for efficiency and precision, utilizing a streamlined pipeline for feature extraction and fusion.

Feature Encoding

The model leverages pre-trained encoders to capture high-quality features from both modalities:

  • Language Encoder: A BERT model extracts word-level embeddings, ensuring that individual tokens (e.g., "black", "right") retain their specific semantic meaning.
  • Visual Encoder: A Darknet-53 backbone processes the image into a grid of visual features.

Grid-Word Attention Module

To resolve modality misalignment, PFOS introduces a bidirectional cross-attention module. This module consists of Language Guide Vision (LGV) and Vision Guide Language (VGL) components, which allow the model to learn nuanced correspondences between specific words and image regions.

Comparison of the traditional anchor-based one-stage method (a) and the proposed PFOS architecture (b) featuring bidirectional guidance.

Anchor-free Localization

In contrast to previous models that refine pre-defined boxes, PFOS employs an anchor-free localization head. It predicts the target object's center point and dimensions directly from the fused multi-modal feature map. This approach eliminates the need for manual anchor tuning and improves the model's robustness to varying object shapes.

The anchor-free localization process, which regresses the center (X, Y) and dimensions (H, W) of the target object.

Experimental Evaluation

PFOS was rigorously tested on standard REC benchmarks, including RefCOCO, RefCOCO+, RefCOCOg, and ReferItGame.

Accuracy and Generalization

The model demonstrated superior performance, achieving 77.13% accuracy on the RefCOCO testB set. Notably, its strong performance on RefCOCO+ highlights its ability to handle appearance-based queries without relying on absolute spatial coordinates. The anchor-free nature of PFOS also ensures better generalization across different datasets compared to "brittle" anchor-based counterparts.

Computational Efficiency

PFOS offers significant advantages in terms of speed and resource usage:

  • Inference Speed: The model operates at approximately 20ms per image, making it roughly 16 times faster than two-stage architectures like MAttNet.
  • Simplicity: By removing anchors and complex proposal stages, PFOS maintains a lower hyperparameter count and a more straightforward training process.

Discussion and Future Work

Visualizations of the attention layers confirm that PFOS successfully performs fine-grained reasoning, correctly highlighting regions associated with specific descriptive adjectives. Despite its strengths, the model faces challenges with expressions requiring Optical Character Recognition (OCR) or involving extremely rare attributes. Future research will explore the integration of OCR modules and more sophisticated reasoning structures to address these edge cases.

Conclusion

PFOS introduces a highly efficient, proposal-free approach to Referring Expression Comprehension. By utilizing bidirectional grid-word attention and direct coordinate regression, the model achieves a new state-of-the-art in both accuracy and speed, providing a robust solution for real-time multi-modal localization tasks.

Summary updated

Abstract

Referring Expression Comprehension (REC) has become one of the most important tasks in visual reasoning, since it is an essential step for many vision-and-language tasks such as visual question answering. However, it has not been widely used in many downstream tasks because it suffers 1) two-stage methods exist heavy computation cost and inevitable error accumulation, and 2) one-stage methods have to depend on lots of hyper-parameters (such as anchors) to generate bounding box. In this paper, we present a proposal-free one-stage (PFOS) model that is able to regress the region-of-interest from the image, based on a textual query, in an end-to-end manner. Instead of using the dominant anchor proposal fashion, we directly take the dense-grid of an image as input for a cross-attention transformer that learns grid-word correspondences. The final bounding box is predicted directly from the image without the time-consuming anchor selection process that previous methods suffer. Our model achieves the state-of-the-art performance on four referring expression datasets with higher efficiency, comparing to previous best one-stage and two-stage methods.
BibTeX
@inproceedings{suo2021proposal,
  author = {Wei Suo},
  title = {{Proposal-free One-stage Referring Expression via Grid-Word Cross-Attention}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/143},
}
Figure