Abstract
Referring Expression Comprehension (ReC) requires localizing a specific object in an image based on a natural language description. While supervised models excel in closed-domain environments, they often fail to generalize to unfamiliar visual domains. We present ReCLIP, a simple yet powerful zero-shot method that repurposes the CLIP model for ReC without additional training. By addressing CLIP's inherent limitations in spatial reasoning and local feature extraction through Isolated Proposal Scoring (IPS) and a rule-based spatial reasoning engine, ReCLIP significantly outperforms previous zero-shot baselines and demonstrates superior robustness on out-of-domain datasets like RefGTA.
Introduction
Referring Expression Comprehension (ReC) is a critical task at the intersection of vision and language, requiring a system to identify a specific target object (e.g., "the man in the blue shirt standing behind the fence") among multiple candidates. While state-of-the-art supervised models achieve high accuracy on standard benchmarks, they are notoriously sensitive to domain shifts. Moving from real-world photography to synthetic imagery or video games often causes performance to collapse, as collecting and labeling exhaustive data for every possible visual domain is economically unfeasible.
This creates a pressing need for zero-shot models that leverage general-purpose knowledge. Large-scale models like CLIP (Contrastive Language-Image Pre-training) are natural candidates; however, they suffer from two primary flaws when applied to ReC: a "global bias" that makes them struggle with small bounding boxes, and a "spatial blindness" regarding the relative positions of objects.
Methodology
ReCLIP addresses these challenges through a two-pronged approach that requires no fine-tuning of the underlying CLIP model.
1. Isolated Proposal Scoring (IPS)
To mitigate CLIP's global bias, ReCLIP employs Isolated Proposal Scoring. Rather than asking CLIP to find an object within a full scene, the system creates modified versions of the image for each candidate bounding box. This is achieved through two complementary techniques:
- Cropping: The image is cropped to the specific bounding box.
- Blurring: The area outside the bounding box is blurred, providing a "visual prompt" that highlights the region of interest while maintaining global context.
These isolated views are then passed to CLIP to determine how well the text description matches the specific region, aligning the ReC task with CLIP’s original pre-training objective of image-text matching.
2. Spatial Relation Resolver
Experimental diagnostics on synthetic datasets (CLEVR) revealed that CLIP performs near random chance on spatial tasks. To solve this, ReCLIP offloads spatial logic to a symbolic engine. The process involves:
- Parsing: The input sentence is converted into a semantic tree using the spaCy dependency parser.
- Decomposition: The system identifies "predicates" (the objects) and "relations" (e.g., "left of", "inside").
- Geometric Heuristics: ReCLIP applies hard-coded geometric rules to resolve these relations. For instance, a "left of" relation is satisfied if the x-coordinate of the center of box A is smaller than that of box B.
The final prediction is an ensemble of CLIP’s visual scores and these deterministic geometric constraints.
Experimental Evaluation
ReCLIP was evaluated across several diverse benchmarks to test both general performance and domain robustness:
- RefCOCO, RefCOCO+, and RefCOCOg: Standard benchmarks based on real-world MS COCO imagery.
- RefGTA: A dataset featuring imagery from Grand Theft Auto V, used to evaluate performance under significant domain shift.
- CLEVR: A synthetic dataset used to isolate and diagnose spatial reasoning capabilities.
Key Insights and Results
ReCLIP establishes a new state-of-the-art for zero-shot Referring Expression Comprehension. Key findings include:
- Zero-Shot Superiority: ReCLIP outperformed the previous leading zero-shot method, CPT, by an absolute margin of 20% across RefCOCO datasets.
- Cross-Domain Robustness: On the RefGTA dataset, ReCLIP achieved approximately 60% accuracy. Notably, this represents an 8% relative improvement over supervised models like UNITER and MDETR. While those models are highly accurate on the data they were trained on, ReCLIP’s generalist nature allows it to maintain performance where specialized models fail.
- The Necessity of Symbolic Logic: The failure of CLIP on the CLEVR dataset underscored that scaling model size does not inherently solve logical or spatial reasoning. The success of ReCLIP’s rule-based resolver suggests that hybrid approaches—combining neural perception with symbolic reasoning—are currently more effective for complex logic.
Limitations and Future Work
While ReCLIP is highly effective, it possesses certain structural limitations. The reliance on a hard-coded dependency parser makes the system brittle; a single grammatical error or complex sentence structure can cause the semantic tree to fail. Furthermore, the current geometric heuristics are limited to 2D space, struggling with 3D concepts like depth ("behind") or ambiguous non-spatial relations (e.g., "the man with the hat").
Future research may focus on replacing these rigid heuristics with flexible, learned modules that preserve the decompositional logic of ReCLIP while offering greater adaptability to linguistic and visual nuances.
Summary updated