Abstract
Visual grounding identifies specific objects in images based on natural language queries. While fully supervised models excel, they require labor-intensive bounding box annotations. This paper introduces Confidence-aware Pseudo-label Learning (CPL), a weakly supervised framework that learns from image-sentence pairs alone. CPL addresses the "heterogeneous gap" by converting visual regions into descriptive pseudo-queries, enabling accurate uni-modal text matching. With a confidence-aware verification module to filter noise, CPL sets new benchmarks across five major datasets, significantly closing the gap between weakly and fully supervised methods.
Introduction
Visual grounding is essential for human-AI interaction, yet the reliance on manually labeled data limits scalability. "Weakly supervised" methods try to learn using only image-level captions, but they face two major hurdles. First is the heterogeneous gap: it is mathematically difficult to directly compare raw pixels to high-level abstract concepts, often leading to "spurious associations." Second is error propagation: if a model makes a mistake early in training by picking the wrong box for a label, it continues to learn from that noisy data.
Methodology
The CPL framework employs a multi-stage pipeline to generate high-quality training signals from noisy data by shifting the problem from cross-modal to uni-modal matching.
- Pseudo-Query Generation: Instead of direct cross-modal matching, CPL describes detected regions using three complementary pipelines: Heuristic+ (spatial/attribute rules), Object-Centric (region-based descriptions via BLIP), and Relation-Aware (holistic image descriptions of interactions).
- Uni-Modal Real Query Propagation: By matching the generated pseudo-text to the real query, the model operates within a single modality (text). This text-to-text matching is significantly more reliable than traditional cross-modal comparison.
- Confidence-aware Verification: To handle imperfect pseudo-labels, the system calculates a confidence score based on image-text matching scores.
- Selective Grounding Loss: During training, the model uses a weighted loss function. If the confidence score for a region-query pair is below a certain threshold, the model ignores it, preventing the "garbage in, garbage out" problem.
Key Results and Evaluation
CPL was evaluated on five widely-used benchmarks: RefCOCO, RefCOCO+, RefCOCOg, ReferItGame, and Flickr30K Entities. It achieved a new state-of-the-art for weakly supervised grounding across all tested datasets:
- Massive Accuracy Gains: On the RefCOCO dataset, CPL reached 70.67% accuracy, a staggering improvement over previous leading methods like ReCLIP (45.78%) and Pseudo-Q (56.02%).
- Efficiency with Small Data: When fine-tuned with just 10% of labeled data, the model outperformed some fully supervised models trained on 100% of the data.
- Robustness: Ablation studies showed that each of the three pseudo-query pipelines contributed significantly to the final performance, proving that linguistic diversity in descriptions is critical.
Discussion and Future Work
The core insight of CPL is that comparing two sentences is much easier for modern AI than comparing a sentence to a patch of pixels. By converting the visual world into "pseudo-labels," the model bypasses the hardest part of the cross-modal gap. Furthermore, the "selective loss" acknowledges that some data is too noisy to be useful, which stabilizes the training process.
However, the method remains dependent on the quality of pre-trained models (like BLIP) and initial object detectors. Future work could involve making the pseudo-label generation "online" (happening during training) or finding ways to generate these labels without relying on heavy pre-trained vision-language models to reduce computational overhead.
Summary updated