Towards Visual Grounding: A Survey

Linhui Xiao, Xiaoshan Yang, Xiangyuan Lan, Yaowei Wang, Changsheng Xu

Abstract

Visual Grounding (VG), also known as Referring Expression Comprehension, involves localizing specific regions within an image based on textual descriptions. This survey tracks the field's evolution over the past decade, from early CNN-LSTM models to modern Grounding Multimodal Large Language Models (GMLLMs). We define key settings, summarize dataset advancements, and highlight the shift toward Generalized Visual Grounding (GVG) to handle real-world complexities like multi-target and no-target scenarios. This comprehensive overview serves as a resource for both beginners and experienced researchers to track the latest research developments.

Bridging the Semantic Gap

Visual Grounding is a fundamental task in multimodal AI that aims to emulate human referential behavior. While humans can effortlessly identify "the man in the white hat and red jacket" in a complex scene, machines must bridge a significant "semantic gap" to align linguistic tokens with visual pixels. This task is critical for applications in robotics, AR/VR, and human-computer interaction.

An illustration of the core visual grounding task: mapping a text query to a specific bounding box.

Historically, research has been hampered by several bottlenecks. Classic benchmarks like RefCOCO are reaching saturation, yet models trained on them often struggle in real-world scenarios. A primary reason is the "single-object assumption"—the expectation that every query refers to exactly one object. Furthermore, a proliferation of experimental settings (weakly supervised, zero-shot, etc.) has led to inconsistent and often unfair comparisons between models.

The Evolutionary Roadmap

The technical history of VG can be divided into three distinct eras:

  1. Preliminary Stage (Pre-2014): Grounding served primarily as a validation tool for Referring Expression Generation (REG) tasks, relying on basic linguistic analysis tools.
  2. Early Stage (2014–2020): This period was dominated by CNNs for vision and LSTMs for text. It saw the emergence of two-stage methods (generating region proposals then matching them to text) and one-stage methods (directly regressing bounding boxes).
  3. Surge Stage (2021–Present): The "Transformer revolution" introduced Visual-Language Pre-trained (VLP) models like CLIP and eventually Grounding Multimodal Large Language Models (GMLLMs). Modern architectures often treat spatial coordinates as "text tokens," allowing models to "read" and "write" locations within a unified language space.

Quantitative analysis reveals a dramatic leap in capability. In 2016, top-tier models achieved approximately 60% accuracy on RefCOCO-testA. By 2024, GMLLMs like Ferret-v2 and OneRef have pushed these scores past 95%, nearing what researchers call "ultimate performance" (~98-99%).

The evolution of paper counts and performance trends over the last decade, showing the rise of GMLLMs and the approach toward the performance ceiling.

As performance on standard benchmarks plateaus, the focus is shifting toward Generalized Visual Grounding (GVG). Unlike traditional Referring Expression Comprehension (REC), GVG requires models to reason about whether a target exists at all (no-target scenarios) or if multiple instances match the description (multi-target scenarios). This shift is supported by new benchmarks like gRefCOCO and Ref-ZOM, which test a model's "common sense" and ability to handle deceptive queries.

Future Directions

Despite reaching high accuracy on legacy datasets, several frontiers remain:

  • Reasoning Complexity: Moving beyond simple descriptions to queries requiring logical reasoning (e.g., "the fruit with the highest Vitamin C content").
  • Dimensional Expansion: Extending 2D grounding to video streams and 3D point clouds is still in its infancy.
  • Universal Grounding: Developing unified frameworks capable of grounding expressions across diverse modalities like speech and gesture in robotics environments.

By standardizing settings and embracing generalized grounding, the field moves closer to achieving truly human-like multimodal comprehension.

Summary updated

Abstract

Visual Grounding is also known as Referring Expression Comprehension and Phrase Grounding. It involves localizing a natural number of specific regions within an image based on a given textual description. The objective of this task is to emulate the prevalent referential relationships in social conversations, equipping machines with human-like multimodal comprehension capabilities. Consequently, it has extensive applications in various domains. However, since 2021, visual grounding has witnessed significant advancements, with emerging new concepts such as grounded pre-training, grounding multimodal LLMs, generalized visual grounding, and giga-pixel grounding, which have brought numerous new challenges. In this survey, we initially examine the developmental history of visual grounding and provide an overview of essential background knowledge. We systematically track and summarize the advancements and meticulously organize the various settings in visual grounding, thereby establishing precise definitions of these settings to standardize future research and ensure a fair comparison. Additionally, we delve into several advanced topics and highlight numerous applications of visual grounding. Finally, we outline the challenges confronting visual grounding and propose valuable directions for future research, which may serve as inspiration for subsequent researchers. By extracting common technical details, this survey encompasses the representative works in each subtopic over the past decade. To the best, this paper presents the most comprehensive overview currently available in the field of grounding. This survey is designed to be suitable for both beginners and experienced researchers, serving as an invaluable resource for understanding key concepts and tracking the latest research developments. We keep tracing related works at https://github.com/linhuixiao/Awesome-Visual-Grounding.
BibTeX
@article{xiao2026towards,
  author = {Linhui Xiao and Xiaoshan Yang and Xiangyuan Lan and Yaowei Wang and Changsheng Xu},
  title = {{Towards Visual Grounding: A Survey}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2026},
  url = {https://openreview.net/forum?id=ikyLAmjmKB},
}
Figure