Look Around Before Locating: Considering Content and Structure Information for Visual Grounding

Shiyi Zheng, Peizhi Zhao, Zhilong Zheng, Peihang He, Haonan Cheng, Yi Cai, Qingbao HuangAAAI 2025 · DOI

TL;DR

This paper introduces SSRVG, a novel transformer-based visual grounding method that tackles the challenge of superficial correlations by explicitly modeling the linguistic structure of referring expressions. By decomposing expressions into "subject" and "context" and using this semi-structured information to guide a multi-branch localization process, SSRVG achieves state-of-the-art performance on several benchmarks.

What Problem Does This Paper Address?

Visual grounding (VG) is the task of locating a specific object in an image based on a natural language description. A major hurdle in this field is that current methods often rely on superficial correlations between words and visual entities, leading to errors when the language is ambiguous or when common entities are involved. For instance, a description like "brown teddy bear printed on a mug" might lead a model to incorrectly focus on any "brown teddy bear" in the image, rather than the specific one on the mug, due to a lack of understanding of the "printed on" relationship. Existing one-stage methods struggle to effectively integrate the sequential nature of text with the spatial nature of images, and while some methods use complex linguistic structures, they often require pre-defined region proposals, limiting their efficiency.

Methods

The proposed method, SSRVG (Semi-Structured Reasoning Visual Grounding), is a transformer-based framework designed to address these limitations. It comprises three main components:

  • Feature Extraction: Utilizes Swin Transformer for visual features and BERT for linguistic features. Crucially, it also leverages the frozen visual encoder of CLIP to obtain cross-modal semantic anchors, which help bridge the semantic gap between modalities and guide alignment.
  • Cross-Modal Content Alignment (CCA): This module acts as the encoder. It aims to build a fine-grained alignment between visual and linguistic features. A key innovation here is the use of CLIP's semantic anchors to correct the alignment process, preventing it from being misled by superficial correlations. It employs a knowledge-guided cross-attention (KGCA) mechanism that incorporates a token-level affinity matrix derived from CLIP's guidance.
  • Multi-Branch Modulated Localization (MML): This module acts as the decoder. It introduces a "semi-structured reasoning" approach by parsing the referring expression into a "subject" and "context" using a dependency parser. This decomposition allows the model to modulate the localization process by first performing rough contextual inference and then refining it with subject-specific information. This is implemented in a cascade form, leveraging a DETR-like decoder.

Datasets

The paper evaluates its method on the following widely-used visual grounding datasets:

  • RefCOCO
  • RefCOCO+
  • RefCOCOg
  • ReferItGame
  • Flickr30K Entities

The authors also mention using a mixed dataset for pre-training the SSRVG-B* model, combining data from RefCOCO/+/g, ReferIt, and Flickr30K.

Key Insights and Ideas

  • Explicit Linguistic Structure is Crucial: The paper strongly advocates for moving beyond simple feature fusion and explicitly modeling the inherent structure within natural language expressions for better visual grounding.
  • CLIP as a Semantic Anchor: Leveraging CLIP's pre-trained visual and textual encoders as a source of semantic anchors is a clever way to guide the alignment process and mitigate the reliance on potentially noisy instance-level annotations.
  • Semi-Structured Reasoning for Ambiguity: Decomposing expressions into "subject" and "context" and using this structured information in a cascaded manner provides a robust way to handle ambiguous descriptions and complex scenes. This allows the model to progressively refine its focus.
  • Balancing Subject and Context: The MML module's design, which considers both subject and context, is shown to be more effective than relying solely on global information or just one of the components. This interaction helps in disambiguating targets.

Key Results

The proposed SSRVG model achieves state-of-the-art performance on several benchmarks. Specifically:

  • SSRVG-B (the largest version) achieves top-1 accuracies of 90.45% on RefCOCOg (val-u), 88.04% on RefCOCO (testB), and 80.14% on RefCOCO+ (testB).
  • The paper demonstrates significant improvements over previous state-of-the-art methods, including a notable improvement of up to 3.13% over HiVG on RefCOCOg.
  • Ablation studies confirm the effectiveness of the individual components: KGCA contributes 1.29% improvement, multi-scale features add 0.74%, and semi-structured reasoning (MML) adds 1.05% over a baseline.

Limitations and Future Work

  • Annotation Quality: The paper notes that the ReferItGame dataset's annotation quality can limit performance, as it's difficult to extract subjects reliably from ungrammatical descriptions.
  • Large Generalist Models: While SSRVG performs competitively, it lags behind very large generalist models on some datasets, likely due to the latter's larger parameter counts and richer training data.
  • Future Work: The authors plan to further investigate approaches for implementing one-stage structured reasoning in open domains to improve generalization.

Summary updated

BibTeX
@inproceedings{zheng2025look,
  author = {Shiyi Zheng and Peizhi Zhao and Zhilong Zheng and Peihang He and Haonan Cheng and Yi Cai and Qingbao Huang},
  title = {{Look Around Before Locating: Considering Content and Structure Information for Visual Grounding}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i2.32158},
}
Figure