Abstract: Referring Expression Comprehension (REC) is a multimodal task at the intersection of computer vision and natural language processing. It aims to uniquely locate a target object in an image based on a natural language description. This review provides a comprehensive taxonomy of REC methods, categorizing them by visual representation granularity: region-based, grid-based, and patch-based. We examine the evolution from complex two-stage pipelines to efficient one-stage Transformer architectures, discuss mainstream benchmarks like RefCOCO, and highlight future challenges in reasoning, interpretability, and real-time performance.
Introduction
Referring Expression Comprehension (REC) bridges the semantic gap between human language and visual content. Unlike standard object detection, which identifies general categories, REC identifies a specific instance based on attributes, spatial relationships, or actions (e.g., "the small white dog sitting to the left of the red car"). This capability is foundational for advanced AI systems, including autonomous robotics, augmented reality, and interactive dialogue systems.
Methodology and Taxonomy
The core challenge of REC is twofold: achieving deep semantic understanding of complex expressions and performing relational reasoning across visual objects. The field has evolved through three primary approaches to visual representation.
Visual Representation Granularity
Methods are categorized by how they "see" and process the image:
- Regional Convolution Granularity (Two-stage): These methods use pre-trained detectors (like Faster R-CNN) to generate object proposals. A second stage then ranks these proposals against the text description. While accurate, they are computationally expensive and limited by the initial detector's performance.
- Grid Convolution Granularity (One-stage): These bypass proposal generation by treating the image as a grid of features. Models like FAOA and RCCF map text directly to spatial coordinates, achieving near real-time speeds (up to 40 FPS).
- Image Patch Granularity (ViT-style): Inspired by Vision Transformers (ViT), these methods divide the image into fixed-size patches. This allows for a unified Transformer architecture where both image patches and text tokens are processed in a shared multimodal space.
Taxonomy of REC Methods
The evolution of REC can be traced through various fusion strategies. Early methods relied on modular networks or attention mechanisms to link modalities. Modern approaches have shifted toward Transformer-based fusion, which allows for dynamic "cross-talk" between visual and linguistic features, leading to superior alignment.
Datasets and Performance
Standard benchmarks include ReferItGame, RefCOCO, RefCOCO+, and RefCOCOg. These vary in complexity; for instance, RefCOCO+ forbids location words, forcing models to rely on appearance, while RefCOCOg features long, complex descriptions.
Key results in the field show a clear trend:
- State-of-the-Art: Models like OFA (One For All) currently lead benchmarks, achieving over 90% accuracy on RefCOCO by leveraging massive multimodal pre-training.
- Efficiency: One-stage models like RCCF have reduced inference time from ~320ms (two-stage) to ~25ms, making them viable for real-world applications.
- Architectural Gains: Transitioning from grid-based to patch-based Transformers (e.g., TransVG++) has yielded accuracy improvements of 5% to 10%.
Key Insights and Future Directions
Despite rapid progress, several challenges remain:
- Reasoning vs. Overfitting: Many models overfit to simple dataset biases. Performance often drops on complex datasets like Cops-Ref, suggesting models still struggle with true multi-step logical reasoning.
- Interpretability: Most models function as "black boxes." Future work must improve visualization of the model's internal reasoning process.
- Dimensional Expansion: The field is expanding beyond 2D images into Video REC (tracking objects over time) and 3D REC (locating objects in point clouds for robotics).
- Weak Supervision: Reducing the reliance on expensive, manually labeled bounding boxes through weakly supervised learning remains a major research goal.
Summary updated