Abstract
Visual Grounding (VG) requires high-quality region-text pairs, yet manual annotation is prohibitively expensive and automated pseudo-labeling often suffers from distributional bias. We present D2AF (Dual-Driven Annotation and Filtering), a framework that shifts the focus from raw data volume to distributional diversity. By combining dual-path annotation, out-of-distribution (OOD) expression expansion, and Gaussian Mixture Model (GMM)-based filtering, D2AF achieves state-of-the-art results across major benchmarks. Our findings demonstrate that prioritizing semantic and spatial diversity over quantity leads to superior generalization in complex, real-world scenarios.
Introduction
Visual Grounding (VG) is the task of localizing specific image regions based on natural language queries. Training robust VG models requires massive amounts of data, but existing automated "pseudo-labeling" pipelines often suffer from a "rich-get-richer" bias, overfitting to common objects and simple descriptions. This results in redundant samples that fail to help models generalize to novel or complex scenarios. D2AF addresses this by focusing on expanding the "distributional coverage" of the dataset, ensuring that the generated data is high-quality, diverse, and balanced.
Methodology
The D2AF framework is designed to generate high-quality pseudo-labels through a three-stage pipeline:
- Dual-Driven Annotation: To ensure both reliability and breadth, D2AF uses two paths. A closed-set path utilizes traditional detectors for high accuracy on known categories, while an open-set path leverages Large Multimodal Models (LMMs) to describe novel concepts and objects outside standard vocabularies.
- OOD Expression Expansion: To prevent linguistic redundancy, the framework uses a Gaussian Mixture Model (GMM) to estimate the density of existing descriptions. It then employs Direct Preference Optimization (DPO) to train the model to generate "out-of-distribution" (OOD) expressions—descriptions that are linguistically unique yet visually accurate.
- Consistency- and Distribution-Aware Filtering:
- Spatial/Semantic Consistency: Generated labels are re-grounded to ensure text-box alignment using IoU and CLIP similarity.
- Distribution Filtering: The GMM is used to prune redundant data. Samples that are too similar to well-represented data are discarded, maintaining a lean and balanced training set.
Experiments and Results
D2AF was evaluated on the gold standards of visual grounding: RefCOCO, RefCOCO+, RefCOCOg, and gRefCOCO. The framework consistently outperformed previous state-of-the-art (SOTA) models across Referring Expression Comprehension (REC), Segmentation (RES), and Generalized Segmentation (GRES) tasks.
Key Insights
- Distribution Over Volume: Adding 1.3 million unfiltered samples yielded only a marginal boost (+0.44%), whereas a filtered set of only 520k samples—specifically chosen to balance the distribution—provided a much larger boost (+2.24%).
- Addressing the "Lazy Annotator": Human-labeled datasets are often biased toward short, efficient captions. D2AF fills the "long-caption" gap, providing complex descriptions that are rare in manual datasets.
- Relative Descriptions: The most substantial gains occurred in "relative descriptions" (e.g., "the person behind the counter"), which are historically difficult for models to learn due to underrepresentation.
Performance Summary
- REC: Average gain of +2.60% over previous SOTA (SimVG).
- RES: Average gain of +2.07% using MaskRIS as a base.
- GRES: Average gain of +2.62% over the ReLA baseline.
Conclusion
D2AF demonstrates that performance gains in visual grounding stem less from the raw quantity of data and more from how effectively that data expands distributional coverage. While the filtering process is computationally intensive, the resulting high-quality dataset significantly improves model robustness. Future work will explore applying this distribution-aware logic to video and 3D visual grounding tasks.
Summary updated