The Solution for the 5th GCAIAC Zero-shot Referring Expression Comprehension Challenge

Yang Yang, Longfei Huang, Feng Yu, Zhihao Guan, Zhonghua Wan, Yang Yang 0128arXiv 2024 · DOI · arXiv

Introduction

Referring Expression Comprehension (REC) is a critical computer vision task that involves locating a specific object within an image based on a natural language description. While large-scale Vision-Language Models (VLMs) like CLIP demonstrate exceptional recognition capabilities, they often suffer from "spatial blindness," struggling to pinpoint exact coordinates in a zero-shot setting where no task-specific training is permitted.

Overview of the REC task: identifying a specific player (B) from a set of candidates (A, B, C) based on a natural language description.

Researchers from the Nanjing University of Science and Technology addressed this limitation to secure first place in the 5th GCAIAC Zero-shot REC Challenge. Their approach moves beyond simple visual prompts by optimizing how models perceive both visual context and textual instructions.

Methodology

The winning strategy utilizes a three-pronged approach to maximize the zero-shot capabilities of the CLIP model:

1. Multi-granularity Visual Prompts

To find the "Goldilocks zone" of visual information, the team combined coarse-grained prompts (such as crops, colorful boxes, and blurred backgrounds) with fine-grained prompts. They leveraged the Segment Anything Model (SAM) to generate precise object masks, allowing the VLM to verify specific attributes against the exact contours of the target object rather than just a bounding box.

2. Text Redundancy Reduction (Hard Denoising)

The researchers discovered that standard prompt templates (e.g., "a photo of [description]") often introduce noise that confuses models in REC tasks. They implemented a "hard denoising" method that strips away redundant textual content, ensuring the model focuses strictly on the identifying features of the target.

The model identifies the specific player in the purple jersey by focusing on key descriptors in the prompt.

3. Joint Prediction via Global Optimization

Rather than treating each object detection as an independent guess, the team employed the Hungarian algorithm to handle images with multiple objects and descriptions. This ensures globally optimal assignments, matching the correct description to the correct bounding box without duplicating results.

Results and Key Insights

The proposed method dominated the challenge leaderboards, achieving a score of 84.825 on Leaderboard A and 71.460 on Leaderboard B. Ablation studies confirmed the value of each component, showing a steady climb from a baseline accuracy of 46.79 to a final offline test accuracy of 72.968.

Method Component Accuracy (ACC)
Baseline (ReCLIP) 46.79
+ Visual Prompts 53.069
+ Removing Redundant Text 60.846
+ Parameter Tuning 63.389
+ Joint Prediction 72.968

Key takeaways from the challenge include the necessity of synergy between coarse and fine prompts—where coarse prompts preserve spatial relationships and fine masks verify attributes—and the counter-intuitive finding that "less is more" when it comes to textual prompts in zero-shot REC.

Conclusion

By combining sophisticated visual segmentation with streamlined text processing and global optimization, this research provides a robust framework for overcoming the spatial limitations of current Vision-Language Models. Future work may focus on reducing the computational overhead of models like SAM and automating the selection of optimal visual prompt combinations.

Summary updated

BibTeX
@misc{yang2024solution,
  author = {Yang Yang and Longfei Huang and Feng Yu and Zhihao Guan and Zhonghua Wan and Yang Yang},
  title = {{The Solution for the 5th GCAIAC Zero-shot Referring Expression Comprehension Challenge}},
  year = {2024},
  eprint = {2407.04998},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.04998},
}
Figure