Introduction
Referring Expression Comprehension (REC), or visual grounding, involves identifying a specific object in an image based on a natural language description. Traditional REC models rely heavily on expensive, manually annotated bounding boxes. Weakly supervised REC aims to learn these mappings using only image-caption pairs. However, current methods often suffer from high computational costs due to two-stage detection pipelines and complex post-processing.
RefCLIP addresses these challenges by redefining REC as an anchor-text matching problem within a one-stage framework. It achieves significant performance improvements while maintaining high inference speeds.
The RefCLIP Framework
RefCLIP shifts the paradigm from ranking pre-extracted regions to matching text with convolutional anchor points.
Anchor-Text Matching
Instead of using a separate region proposal network, RefCLIP utilizes a one-stage detector (e.g., YOLOv3). It treats anchor points in the feature maps as potential matches for the input text. To manage the high number of anchors, RefCLIP employs two filtering strategies:
- Scale Selection: Focuses on specific grid scales (e.g., 13x13) where objects are most likely to reside.
- Confidence Filtering: Retains only the top 10% of anchors based on visual confidence scores to reduce background noise.
Contrastive Learning
The model is trained using an anchor-based contrastive loss (InfoNCE). It maximizes the similarity between a text description and its corresponding visual anchor while minimizing similarity with negative anchors from the same or different images. This approach leverages the large receptive fields of deep convolutional layers, which provide sufficient context for grounding without explicit region extraction.
Weakly-Supervised Training Scheme
RefCLIP functions as a "universal teacher" in a model-agnostic training pipeline. It generates pseudo-labels (predicted bounding boxes) for unlabeled data, which are then used to supervise "student" models—high-performance REC architectures like TransVG or SimREC.
To mitigate the impact of noisy pseudo-labels, the framework incorporates:
- Exponential Moving Average (EMA): Stabilizes the learning process.
- Data Augmentation: Enhances the robustness of student models against teacher errors.
Key Results
RefCLIP demonstrates state-of-the-art performance across multiple benchmarks, including RefCOCO, RefCOCO+, RefCOCOg, and ReferItGame.
- Accuracy: On RefCOCO, RefCLIP improved performance by 24.87% over previous weakly supervised methods.
- Efficiency: Operating at 31.3 FPS, RefCLIP is approximately five times faster than traditional two-stage weakly supervised models.
- Generalization: As a teacher, RefCLIP successfully boosted the performance of various student architectures, proving its utility as a universal supervision source.
Conclusion
RefCLIP offers a fast and effective solution for weakly supervised REC. By simplifying the task to anchor-text matching and serving as a universal teacher, it bridges the gap between weakly supervised learning and high-performance grounding architectures. Future work will explore its application to more complex tasks like referring expression segmentation and video grounding.
Summary updated