Abstract
Visual Grounding (VG), also known as Referring Expression Comprehension, involves localizing specific regions within an image based on textual descriptions. This survey tracks the field's evolution over the past decade, from early CNN-LSTM models to modern Grounding Multimodal Large Language Models (GMLLMs). We define key settings, summarize dataset advancements, and highlight the shift toward Generalized Visual Grounding (GVG) to handle real-world complexities like multi-target and no-target scenarios. This comprehensive overview serves as a resource for both beginners and experienced researchers to track the latest research developments.
Bridging the Semantic Gap
Visual Grounding is a fundamental task in multimodal AI that aims to emulate human referential behavior. While humans can effortlessly identify "the man in the white hat and red jacket" in a complex scene, machines must bridge a significant "semantic gap" to align linguistic tokens with visual pixels. This task is critical for applications in robotics, AR/VR, and human-computer interaction.
Historically, research has been hampered by several bottlenecks. Classic benchmarks like RefCOCO are reaching saturation, yet models trained on them often struggle in real-world scenarios. A primary reason is the "single-object assumption"—the expectation that every query refers to exactly one object. Furthermore, a proliferation of experimental settings (weakly supervised, zero-shot, etc.) has led to inconsistent and often unfair comparisons between models.
The Evolutionary Roadmap
The technical history of VG can be divided into three distinct eras:
- Preliminary Stage (Pre-2014): Grounding served primarily as a validation tool for Referring Expression Generation (REG) tasks, relying on basic linguistic analysis tools.
- Early Stage (2014–2020): This period was dominated by CNNs for vision and LSTMs for text. It saw the emergence of two-stage methods (generating region proposals then matching them to text) and one-stage methods (directly regressing bounding boxes).
- Surge Stage (2021–Present): The "Transformer revolution" introduced Visual-Language Pre-trained (VLP) models like CLIP and eventually Grounding Multimodal Large Language Models (GMLLMs). Modern architectures often treat spatial coordinates as "text tokens," allowing models to "read" and "write" locations within a unified language space.
Performance Trends and the "Ultimate Performance" Ceiling
Quantitative analysis reveals a dramatic leap in capability. In 2016, top-tier models achieved approximately 60% accuracy on RefCOCO-testA. By 2024, GMLLMs like Ferret-v2 and OneRef have pushed these scores past 95%, nearing what researchers call "ultimate performance" (~98-99%).
As performance on standard benchmarks plateaus, the focus is shifting toward Generalized Visual Grounding (GVG). Unlike traditional Referring Expression Comprehension (REC), GVG requires models to reason about whether a target exists at all (no-target scenarios) or if multiple instances match the description (multi-target scenarios). This shift is supported by new benchmarks like gRefCOCO and Ref-ZOM, which test a model's "common sense" and ability to handle deceptive queries.
Future Directions
Despite reaching high accuracy on legacy datasets, several frontiers remain:
- Reasoning Complexity: Moving beyond simple descriptions to queries requiring logical reasoning (e.g., "the fruit with the highest Vitamin C content").
- Dimensional Expansion: Extending 2D grounding to video streams and 3D point clouds is still in its infancy.
- Universal Grounding: Developing unified frameworks capable of grounding expressions across diverse modalities like speech and gesture in robotics environments.
By standardizing settings and embracing generalized grounding, the field moves closer to achieving truly human-like multimodal comprehension.
Summary updated