Introduction
Referring Expression Comprehension (REC) is a critical computer vision task that involves locating a specific object within an image based on a natural language description. While large-scale Vision-Language Models (VLMs) like CLIP demonstrate exceptional recognition capabilities, they often suffer from "spatial blindness," struggling to pinpoint exact coordinates in a zero-shot setting where no task-specific training is permitted.
Researchers from the Nanjing University of Science and Technology addressed this limitation to secure first place in the 5th GCAIAC Zero-shot REC Challenge. Their approach moves beyond simple visual prompts by optimizing how models perceive both visual context and textual instructions.
Methodology
The winning strategy utilizes a three-pronged approach to maximize the zero-shot capabilities of the CLIP model:
1. Multi-granularity Visual Prompts
To find the "Goldilocks zone" of visual information, the team combined coarse-grained prompts (such as crops, colorful boxes, and blurred backgrounds) with fine-grained prompts. They leveraged the Segment Anything Model (SAM) to generate precise object masks, allowing the VLM to verify specific attributes against the exact contours of the target object rather than just a bounding box.
2. Text Redundancy Reduction (Hard Denoising)
The researchers discovered that standard prompt templates (e.g., "a photo of [description]") often introduce noise that confuses models in REC tasks. They implemented a "hard denoising" method that strips away redundant textual content, ensuring the model focuses strictly on the identifying features of the target.
3. Joint Prediction via Global Optimization
Rather than treating each object detection as an independent guess, the team employed the Hungarian algorithm to handle images with multiple objects and descriptions. This ensures globally optimal assignments, matching the correct description to the correct bounding box without duplicating results.
Results and Key Insights
The proposed method dominated the challenge leaderboards, achieving a score of 84.825 on Leaderboard A and 71.460 on Leaderboard B. Ablation studies confirmed the value of each component, showing a steady climb from a baseline accuracy of 46.79 to a final offline test accuracy of 72.968.
| Method Component | Accuracy (ACC) |
|---|---|
| Baseline (ReCLIP) | 46.79 |
| + Visual Prompts | 53.069 |
| + Removing Redundant Text | 60.846 |
| + Parameter Tuning | 63.389 |
| + Joint Prediction | 72.968 |
Key takeaways from the challenge include the necessity of synergy between coarse and fine prompts—where coarse prompts preserve spatial relationships and fine masks verify attributes—and the counter-intuitive finding that "less is more" when it comes to textual prompts in zero-shot REC.
Conclusion
By combining sophisticated visual segmentation with streamlined text processing and global optimization, this research provides a robust framework for overcoming the spatial limitations of current Vision-Language Models. Future work may focus on reducing the computational overhead of models like SAM and automating the selection of optimal visual prompt combinations.
Summary updated