Confidence-aware Pseudo-label Learning for Weakly Supervised Visual Grounding

Yang Liu, Jiahua Zhang, Qingchao Chen, Yuxin Peng

Abstract

Visual grounding identifies specific objects in images based on natural language queries. While fully supervised models excel, they require labor-intensive bounding box annotations. This paper introduces Confidence-aware Pseudo-label Learning (CPL), a weakly supervised framework that learns from image-sentence pairs alone. CPL addresses the "heterogeneous gap" by converting visual regions into descriptive pseudo-queries, enabling accurate uni-modal text matching. With a confidence-aware verification module to filter noise, CPL sets new benchmarks across five major datasets, significantly closing the gap between weakly and fully supervised methods.

Introduction

Visual grounding is essential for human-AI interaction, yet the reliance on manually labeled data limits scalability. "Weakly supervised" methods try to learn using only image-level captions, but they face two major hurdles. First is the heterogeneous gap: it is mathematically difficult to directly compare raw pixels to high-level abstract concepts, often leading to "spurious associations." Second is error propagation: if a model makes a mistake early in training by picking the wrong box for a label, it continues to learn from that noisy data.

Comparison of existing cross-modal methods vs. the proposed uni-modal approach.

Methodology

The CPL framework employs a multi-stage pipeline to generate high-quality training signals from noisy data by shifting the problem from cross-modal to uni-modal matching.

  1. Pseudo-Query Generation: Instead of direct cross-modal matching, CPL describes detected regions using three complementary pipelines: Heuristic+ (spatial/attribute rules), Object-Centric (region-based descriptions via BLIP), and Relation-Aware (holistic image descriptions of interactions).
  2. Uni-Modal Real Query Propagation: By matching the generated pseudo-text to the real query, the model operates within a single modality (text). This text-to-text matching is significantly more reliable than traditional cross-modal comparison.
  3. Confidence-aware Verification: To handle imperfect pseudo-labels, the system calculates a confidence score based on image-text matching scores.
  4. Selective Grounding Loss: During training, the model uses a weighted loss function. If the confidence score for a region-query pair is below a certain threshold, the model ignores it, preventing the "garbage in, garbage out" problem.
The CPL framework overview, including pseudo-query generation and real query propagation.

Key Results and Evaluation

CPL was evaluated on five widely-used benchmarks: RefCOCO, RefCOCO+, RefCOCOg, ReferItGame, and Flickr30K Entities. It achieved a new state-of-the-art for weakly supervised grounding across all tested datasets:

  • Massive Accuracy Gains: On the RefCOCO dataset, CPL reached 70.67% accuracy, a staggering improvement over previous leading methods like ReCLIP (45.78%) and Pseudo-Q (56.02%).
  • Efficiency with Small Data: When fine-tuned with just 10% of labeled data, the model outperformed some fully supervised models trained on 100% of the data.
  • Robustness: Ablation studies showed that each of the three pseudo-query pipelines contributed significantly to the final performance, proving that linguistic diversity in descriptions is critical.

Discussion and Future Work

The core insight of CPL is that comparing two sentences is much easier for modern AI than comparing a sentence to a patch of pixels. By converting the visual world into "pseudo-labels," the model bypasses the hardest part of the cross-modal gap. Furthermore, the "selective loss" acknowledges that some data is too noisy to be useful, which stabilizes the training process.

However, the method remains dependent on the quality of pre-trained models (like BLIP) and initial object detectors. Future work could involve making the pseudo-label generation "online" (happening during training) or finding ways to generate these labels without relying on heavy pre-trained vision-language models to reduce computational overhead.

Summary updated

Abstract

Visual grounding aims at localizing the target object in image which is most related to the given free-form natural language query. As labeling the position of target object is labor-intensive, the weakly supervised methods, where only image-sentence annotations are required during model training have recently received increasing attention. Most of the existing weakly-supervised methods first generate region proposals via pre-trained object detectors and then employ either cross-modal similarity score or reconstruction loss as the criteria to select proposal from them. However, due to the cross-modal heterogeneous gap, these method often suffer from high confidence spurious association and model prone to error propagation. In this paper, we propose Confidence-aware Pseudo-label Learning (CPL) to overcome the above limitations. Specifically, we first adopt both the uni-modal and cross-modal pre-trained models and propose conditional prompt engineering to automatically generate multiple ‘descriptive, realistic and diverse’ pseudo language queries for each region proposal, and then establish reliable cross-modal association for model training based on the uni-modal similarity score (between pseudo and real text queries). Secondly, we propose a confidence-aware pseudo label verification module which reduces the amount of noise encountered in the training process and the risk of error propagation. Experiments on five widely used datasets validate the efficacy of our proposed components and demonstrate state-of-the-art performance. Code can be found at https://github.com/zjh31/CPL.git
BibTeX
@inproceedings{liu2023confidence,
  author = {Yang Liu and Jiahua Zhang and Qingchao Chen and Yuxin Peng},
  title = {{Confidence-aware Pseudo-label Learning for Weakly Supervised Visual Grounding}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.00265},
}
Figure