Abstract
Referring Expression Comprehension (REC) is a fundamental task in computer vision and natural language processing that involves localizing a target object in an image based on a linguistic description. Traditional one-stage methods often rely on pre-defined anchors, which can lead to modality misalignment and reduced generalization across datasets. This paper presents PFOS (Proposal-Free One-Stage), a framework that eliminates anchors in favor of a bidirectional grid-word cross-attention mechanism. By directly regressing object coordinates from multi-modal feature maps, PFOS achieves state-of-the-art accuracy while significantly outperforming existing methods in inference speed.
Introduction
The goal of REC is to bridge the gap between visual perception and natural language understanding. Historically, the field has been dominated by two-stage methods that generate region proposals and then rank them, or one-stage methods that utilize anchors. However, two-stage approaches are computationally intensive, and anchor-based one-stage models often struggle with the fine-grained details of complex queries. The proposed PFOS model addresses these challenges by removing the anchor dependency and implementing a more effective modality fusion strategy.
Methodology
The PFOS architecture is designed for efficiency and precision, utilizing a streamlined pipeline for feature extraction and fusion.
Feature Encoding
The model leverages pre-trained encoders to capture high-quality features from both modalities:
- Language Encoder: A BERT model extracts word-level embeddings, ensuring that individual tokens (e.g., "black", "right") retain their specific semantic meaning.
- Visual Encoder: A Darknet-53 backbone processes the image into a grid of visual features.
Grid-Word Attention Module
To resolve modality misalignment, PFOS introduces a bidirectional cross-attention module. This module consists of Language Guide Vision (LGV) and Vision Guide Language (VGL) components, which allow the model to learn nuanced correspondences between specific words and image regions.
Anchor-free Localization
In contrast to previous models that refine pre-defined boxes, PFOS employs an anchor-free localization head. It predicts the target object's center point and dimensions directly from the fused multi-modal feature map. This approach eliminates the need for manual anchor tuning and improves the model's robustness to varying object shapes.
Experimental Evaluation
PFOS was rigorously tested on standard REC benchmarks, including RefCOCO, RefCOCO+, RefCOCOg, and ReferItGame.
Accuracy and Generalization
The model demonstrated superior performance, achieving 77.13% accuracy on the RefCOCO testB set. Notably, its strong performance on RefCOCO+ highlights its ability to handle appearance-based queries without relying on absolute spatial coordinates. The anchor-free nature of PFOS also ensures better generalization across different datasets compared to "brittle" anchor-based counterparts.
Computational Efficiency
PFOS offers significant advantages in terms of speed and resource usage:
- Inference Speed: The model operates at approximately 20ms per image, making it roughly 16 times faster than two-stage architectures like MAttNet.
- Simplicity: By removing anchors and complex proposal stages, PFOS maintains a lower hyperparameter count and a more straightforward training process.
Discussion and Future Work
Visualizations of the attention layers confirm that PFOS successfully performs fine-grained reasoning, correctly highlighting regions associated with specific descriptive adjectives. Despite its strengths, the model faces challenges with expressions requiring Optical Character Recognition (OCR) or involving extremely rare attributes. Future research will explore the integration of OCR modules and more sophisticated reasoning structures to address these edge cases.
Conclusion
PFOS introduces a highly efficient, proposal-free approach to Referring Expression Comprehension. By utilizing bidirectional grid-word attention and direct coordinate regression, the model achieves a new state-of-the-art in both accuracy and speed, providing a robust solution for real-time multi-modal localization tasks.
Summary updated