Abstract
Unsupervised visual grounding methods alleviate the issue of expensive manual annotation of image-query pairs by generating pseudo-queries. However, existing methods are prone to confusing the spatial relationships between objects and rely on designing complex prompt modules to generate query texts, which severely impedes the ability to generate accurate and comprehensive queries due to ambiguous spatial relationships and manually-defined fixed templates. To tackle these challenges, we propose an omni-directional language query generation approach for unsupervised visual grounding named Omni-Q. Specifically, we develop a 3D spatial relation module to extend the 2D spatial representation to 3D, thereby utilizing 3D location information to accurately determine the spatial position among objects. Besides, we introduce a spatial graph module, leveraging the power of graph structures to establish accurate and diverse object relationships and thus enhancing the flexibility of query generation. Extensive experiments on five public benchmark datasets demonstrate that our method significantly outperforms existing state-of-the-art unsupervised methods by up to 16.17%. In addition, when applied in the supervised setting, our method can freely save up to 60% human annotations without a loss of performance.
Introduction
Visual grounding—the task of locating a specific object in an image based on a natural language query—is a fundamental challenge in multi-modal AI. Traditional supervised training requires massive datasets of manually annotated image-query pairs, which is both time-consuming and costly. Unsupervised methods attempt to bridge this gap by automatically generating pseudo-queries. However, current state-of-the-art unsupervised approaches, such as Pseudo-Q, face two significant hurdles: 2D spatial ambiguity and rigid text templates. By relying on 2D coordinates, these models often misinterpret depth-based relationships (e.g., confusing "behind" with "below"). Furthermore, fixed templates limit the diversity and naturalness of the generated queries.
To overcome these limitations, we introduce Omni-Q, a framework that shifts from 2D to 3D spatial reasoning and replaces rigid templates with a flexible graph-based generation process.
The Omni-Q Framework
Omni-Q decomposes the grounding process into three primary modules designed to capture the richness and depth of visual scenes.
Object Perception Module
Rather than relying on simple object detectors that only provide class labels, Omni-Q utilizes a pre-trained object understanding model (GRiT). This module extracts bounding boxes alongside rich, natural language descriptions for every detected object, providing a semantic foundation for query generation.
3D Spatial Relation Module
The core innovation of Omni-Q is the transition from 2D to 3D spatial representation. By incorporating a depth estimation model, the system projects 2D image data into a 3D space. This allows the model to decouple relative and absolute positions, accurately distinguishing between objects that are horizontally adjacent versus those separated by depth (e.g., "in front of" vs. "behind").
Spatial Graph Module
To move beyond repetitive templates, Omni-Q constructs a directed spatial graph where objects serve as nodes and their 3D relationships define the edges. By performing random walks across this graph, the model can generate diverse, multi-object descriptions that reflect the complex interconnections of real-world scenes. This graph-based approach significantly enhances the flexibility and naturalness of the resulting pseudo-queries.
Experimental Evaluation
We evaluated Omni-Q on five standard visual grounding benchmarks: RefCOCO, RefCOCO+, RefCOCOg, ReferItGame, and Flickr30K Entities. These datasets cover a wide range of challenges, from complex spatial reasoning to detailed object attribute descriptions.
Key Results and Insights
Omni-Q establishes a new state-of-the-art for unsupervised visual grounding. On the RefCOCOg dataset, it achieved a top-1 accuracy of 65.99%, representing a 16.17% improvement over previous methods. Similar gains were observed on ReferItGame (+9.59%), highlighting the model's superior spatial reasoning capabilities.
A particularly striking result is the "40% Rule." As shown in the performance charts, Omni-Q can match the performance of a fully supervised model while using only 40% of the human-labeled data in a semi-supervised setting. This demonstrates that high-quality synthetic data can effectively substitute for a majority of manual annotations without sacrificing performance.
Conclusion
Omni-Q demonstrates that 3D spatial reasoning and graph-based query generation are critical for advancing unsupervised visual grounding. By resolving 2D ambiguities and moving away from rigid templates, Omni-Q generates more accurate and diverse training data. While the system currently depends on the quality of upstream depth and perception models, it provides a robust foundation for future work in incorporating deeper semantic contexts, such as object interactions and actions, into scene understanding.
Summary updated