RefCLIP: A Universal Teacher for Weakly Supervised Referring Expression Comprehension

Lei Jin, Gen Luo, Yiyi Zhou, Xiaoshuai Sun, Guannan Jiang, Annan Shu, Rongrong Ji

Introduction

Referring Expression Comprehension (REC), or visual grounding, involves identifying a specific object in an image based on a natural language description. Traditional REC models rely heavily on expensive, manually annotated bounding boxes. Weakly supervised REC aims to learn these mappings using only image-caption pairs. However, current methods often suffer from high computational costs due to two-stage detection pipelines and complex post-processing.

RefCLIP addresses these challenges by redefining REC as an anchor-text matching problem within a one-stage framework. It achieves significant performance improvements while maintaining high inference speeds.

The RefCLIP Framework

RefCLIP shifts the paradigm from ranking pre-extracted regions to matching text with convolutional anchor points.

Overview of RefCLIP and its weakly-supervised training scheme.

Anchor-Text Matching

Instead of using a separate region proposal network, RefCLIP utilizes a one-stage detector (e.g., YOLOv3). It treats anchor points in the feature maps as potential matches for the input text. To manage the high number of anchors, RefCLIP employs two filtering strategies:

  1. Scale Selection: Focuses on specific grid scales (e.g., 13x13) where objects are most likely to reside.
  2. Confidence Filtering: Retains only the top 10% of anchors based on visual confidence scores to reduce background noise.

Contrastive Learning

The model is trained using an anchor-based contrastive loss (InfoNCE). It maximizes the similarity between a text description and its corresponding visual anchor while minimizing similarity with negative anchors from the same or different images. This approach leverages the large receptive fields of deep convolutional layers, which provide sufficient context for grounding without explicit region extraction.

Weakly-Supervised Training Scheme

RefCLIP functions as a "universal teacher" in a model-agnostic training pipeline. It generates pseudo-labels (predicted bounding boxes) for unlabeled data, which are then used to supervise "student" models—high-performance REC architectures like TransVG or SimREC.

To mitigate the impact of noisy pseudo-labels, the framework incorporates:

  • Exponential Moving Average (EMA): Stabilizes the learning process.
  • Data Augmentation: Enhances the robustness of student models against teacher errors.

Key Results

RefCLIP demonstrates state-of-the-art performance across multiple benchmarks, including RefCOCO, RefCOCO+, RefCOCOg, and ReferItGame.

Ablation studies on anchor selection and performance curves on Test A and Test B.
  • Accuracy: On RefCOCO, RefCLIP improved performance by 24.87% over previous weakly supervised methods.
  • Efficiency: Operating at 31.3 FPS, RefCLIP is approximately five times faster than traditional two-stage weakly supervised models.
  • Generalization: As a teacher, RefCLIP successfully boosted the performance of various student architectures, proving its utility as a universal supervision source.

Conclusion

RefCLIP offers a fast and effective solution for weakly supervised REC. By simplifying the task to anchor-text matching and serving as a universal teacher, it bridges the gap between weakly supervised learning and high-performance grounding architectures. Future work will explore its application to more complex tasks like referring expression segmentation and video grounding.

Summary updated

Abstract

Referring Expression Comprehension (REC) is a task of grounding the referent based on an expression, and its development is greatly limited by expensive instance-level annotations. Most existing weakly supervised methods are built based on two-stage detection networks, which are computationally expensive. In this paper, we resort to the efficient one-stage detector and propose a novel weakly supervised model called RefCLIP.Specifically, RefCLIP redefines weakly supervised REC as an anchor-text matching problem, which can avoid the complex post-processing in existing methods. To achieve weakly supervised learning, we introduce anchor-based contrastive loss to optimize Re-fCLlP via numerous anchor-text pairs. Based on RefCLIP, we further propose the first model-agnostic weakly supervised training scheme for existing REC models, where RefCLIP acts as a mature teacher to generate pseudo-labels for teaching common REC models. With our careful designs, this scheme can even help existing REC models achieve better weakly supervised performance than RefCLIP, e.g., TransVg and SimREC. To validate our approaches, we conduct extensive experiments on four REC benchmarks, i.e., RefCOCO, RefCOCO+, RefCOCOg and ReferItGame. Experimental results not only report our significant performance gains over existing weakly supervised models, e.g., +24.87% on RefCOCO, but also show the 5x faster inference speed. Project: https://RefCLIP.github.io.
BibTeX
@inproceedings{jin2023refclip,
  author = {Lei Jin and Gen Luo and Yiyi Zhou and Xiaoshuai Sun and Guannan Jiang and Annan Shu and Rongrong Ji},
  title = {{RefCLIP: A Universal Teacher for Weakly Supervised Referring Expression Comprehension}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.00263},
}
Figure