TL;DR
This paper provides a comprehensive survey of the last decade of research in visual grounding, charting its evolution from early CNN-LSTM pipelines to modern Grounding Multimodal Large Language Models (GMLLMs). It standardizes confusing experimental settings, analyzes performance bottlenecks on classic benchmarks, and introduces emerging paradigms like Generalized Visual Grounding.
What Problem Does This Paper Address?
Visual Grounding (VG)—also known as Referring Expression Comprehension (REC) or Phrase Grounding (PG)—bridges the semantic gap between vision and language by localizing specific regions in an image based on a textual description.
While the field has witnessed massive breakthroughs since 2021 due to the rise of Vision-Language Pre-training (VLP) and Large Language Models, this rapid expansion has introduced significant fragmentation. The authors address several critical gaps in the current literature:
- Confusing Experimental Settings: Boundaries between fully supervised, weakly supervised, unsupervised, and zero-shot settings have become blurred, leading to unfair model comparisons.
- Benchmark Saturation: Standard datasets (like RefCOCO) are nearly a decade old and hitting performance ceilings, yet the community lacks a clear trajectory for future benchmark development.
- Brittle Assumptions: Traditional visual grounding operates on the rigid assumption that exactly one target object exists per text query, which fails in complex, real-world scenarios.
Methods
The survey systematically categorizes the technical roadmaps and structural frameworks that define the field:
1. Technical Roadmaps
- Traditional CNN-Based: Early methods combining CNNs for image feature extraction and RNNs/LSTMs for text encoding, mapping them into a shared vector space using two-stage (proposal-based) or one-stage (anchor-based) pipelines.
- Transformer-Based: Utilizing architectures like ViT and DETR to reformulate grounding as a direct coordinate regression problem.
- VLP-Based Transfer: Leveraging large-scale pre-trained models like CLIP to exploit inherently aligned cross-modal feature spaces.
- Grounding-Oriented Pre-training: Models explicitly trained on region-phrase correspondences (e.g., GLIP, MDETR).
- Grounding Multimodal LLMs (GMLLMs): The latest paradigm, which treats bounding box coordinates as textual tokens, enabling autoregressive generation of both text and spatial locations within a unified dialogue framework.
2. Structural Frameworks
The authors classify model architectures into five distinct structural types: 2+1 structure (separate encoders + fusion encoder), 2+2 structure (encoder-decoder with query anchors), Two-encoder structure (no explicit fusion module), One-tower structure (modality-shared feature spaces), and GMLLM structure (visual features mapped directly into an LLM's token space).
3. Generalized Visual Grounding (GVG)
The survey formalizes GVG, a paradigm shift that breaks the "one-to-one" constraint to handle three realistic scenarios: grounding a unique target, grounding multiple targets, or handling queries where no target is present.
Datasets
The paper compiles and categorizes dozens of datasets across different eras:
- Classical Benchmarks: RefCOCO, RefCOCO+, RefCOCOg, ReferItGame, and Flickr30k Entities.
- Generalized VG Benchmarks: gRefCOCO, Ref-ZOM, $D^3$, and RefDrone.
- GMLLM-Specific Datasets: GRIT, HC-RefLoCo, HumanRef, GVC, and Ref-L4.
- Universal/Large-Scale Scenarios: Grounding-100M (100 million samples), GigaGround (gigapixel-scale), and MGrounding-630K (multi-image context).
Key Insights and Ideas
- The Ultimate Performance Boundary: The authors conducted a unique experiment to find the upper performance limit of classic benchmarks. By training a state-of-the-art model on the entirety of the RefCOCO/+/g datasets (combining train, validation, and test splits), they revealed that data errors, label noise, and inherent ambiguities cap performance at around 98-99%. Current models are already hitting 92-95%, meaning these benchmarks are effectively saturated.
- Coordinates as Vocabulary: One of the most elegant transitions in the field is how GMLLMs handle spatial data. Instead of using specialized regression heads, modern models discretize coordinates into spatial bins and treat them as words in a dictionary, unifying localization and natural language generation.
- The Necessity of GVG: Traditional evaluation fails when a user queries something not present in the image. GVG introduces metrics like "No-target accuracy" (N-acc) and Precision@F1 to ensure models can safely say "not found" rather than forcing a false positive prediction.
Key Results
The survey tracks a clear trajectory of performance growth over the past decade:
- Early CNN-based architectures (2016–2020) hovered between 60% and 80% accuracy on the RefCOCO-testA benchmark.
- The introduction of Transformers and VLP models (2021–2023) pushed accuracies into the mid-to-high 80s.
- Modern GMLLMs (such as Ferret-v2 and LION) and advanced one-tower models (OneRef-L) dominate the upper echelons, achieving between 92% and 95.4% accuracy, rapidly closing the gap to the theoretical dataset ceiling.
Limitations and Future Work
The authors identify several open challenges and future directions:
- Textual Reasoning Bottlenecks: Most current datasets rely on simple, descriptive phrases. Future benchmarks must demand complex logical reasoning (e.g., "ground the fruit richest in vitamin C").
- Transition to Video: Grounding in static images is mature, but video object grounding—which requires modeling temporal consistency, tracking through occlusions, and handling motion blur—remains an open frontier.
- Data Scaling Issues: While contrastive image-text datasets span billions of samples (e.g., LAION), high-quality, fine-grained bounding box triplet data is incredibly scarce and expensive to annotate. Developing self-supervised grounding pre-training methods (like Masked Referring Modeling) is essential to scale further.
Summary updated