TL;DR
Retraining large-scale vision-language retrievers like CLIP when query distributions shift (due to language, length, or multimodal inputs) is prohibitively expensive because it requires re-indexing billions of database items. GRAPE (Grouped Ranking-Aware Policy Optimization Enhancement) keeps the retriever frozen and trains an LLM query rewriter using Group Relative Policy Optimization (GRPO) guided by target item rankings. By substituting naive cosine similarity rewards with a corpus-relative ranking reward, GRAPE eliminates the "score inflation" trap and delivers an average +4.9% boost in Recall@10 across diverse retrieval benchmarks.
What Problem Does This Paper Address?
CLIP models underpin many real-world, billion-scale visual retrieval engines. However, CLIP's zero-shot retrieval accuracy degrades when user queries deviate from its original training distribution—such as queries in non-English languages, long-form paragraphs, or multimodal queries (image + text instructions).
While fine-tuning the retriever on new domains can improve performance, it modifies the latent embedding space. This invalidates pre-computed vector databases, forcing costly re-embedding and system redeployment. A common alternative is using a Large Language Model (LLM) at query time to rewrite downstream queries into retriever-friendly text. However, zero-shot LLMs lack awareness of CLIP’s specific embedding behavior. Furthermore, naively fine-tuning the LLM using CLIP similarity scores causes score inflation: the LLM learns to generate generic, non-discriminative buzzwords (e.g., "real-world environment," "atmosphere") that inflate similarity scores across all corpus items while degrading true retrieval rankings.
Methods
GRAPE introduces an online reinforcement learning framework that fine-tunes an LLM query rewriter while keeping the downstream retriever and database embeddings completely frozen:
- Constrained Query Generation: Given a query $q$, the LLM policy generates $K$ candidate rewrites formatted with
<think>...</think>reasoning chains and<answer>...</answer>outputs. - Frozen Retrieval Evaluation: Each valid rewrite is embedded by the frozen CLIP text encoder and scored against the candidate image pool to determine the rank $r_k \in \{1, \dots, N\}$ of the target ground-truth item.
- Dual Reward Structure:
- Format Reward ($R^f$): Rewards outputs that adhere to format tags ($+1$) and penalizes malformed responses ($-1$, skipping retrieval).
- Ranking-Based Reward ($R^r$): Linearly scales the rank position: $$R_k^r = 1 - \frac{2(r_k - 1)}{N - 1}$$
- Group Relative Policy Optimization (GRPO): Advantage estimation is computed by normalizing total rewards ($R_k = R_k^f + R_k^r$) across the $K$ candidates for that query, regularized by a KL divergence penalty against the reference model. Because GRPO's group advantage normalization is affine-invariant, any linear scaling of the ranking reward reduces directly to the relative ranking difference, guiding optimization toward true visual discrimination rather than raw similarity score hacking.
For text-only benchmarks, the authors use Qwen2.5-3B-Instruct; for multimodal tasks, they employ Qwen2-VL-7B.
Datasets
The authors evaluated GRAPE across five benchmarks addressing three distinct distribution shifts:
- Multilingual Shifts:
- Flickr30k-CN (Chinese image-text retrieval)
- CVLUE (Chinese vision-language understanding)
- XM3600 (massively multilingual benchmark spanning 36 languages)
- Length Shifts:
- Wikipedia (English text-to-image retrieval using long, descriptive paragraphs)
- Modality Shifts:
- CIRR (Composed Image Retrieval on Real-life images; query consists of a reference image plus modification text)
No new datasets were introduced.
Key Insights and Ideas
- The Score Inflation Trap: Optimizing query rewrites directly against CLIP cosine similarity rewards backfires. The LLM exploits high-frequency semantic associations, peppering rewrites with generic descriptors that elevate similarity across both targets and distractors alike, causing Recall@1 to steadily drop during training.
- Affine Invariance of Relative Advantages: The authors prove that normalizing rewards within a group in GRPO renders affine-mapped ranking rewards invariant to scale and offset. The driving policy signal becomes purely the relative rank order between rewrites.
- Distinct Adaptation Mechanisms:
- Cross-lingual: Recovers implicit semantic entities omitted during direct translation (e.g., inferring "baseball player" instead of just literal "pitching").
- Long-text: Distills multi-hundred-word documents down to visually salient anchor words while purging conversational filler.
- Multimodal fusion: Translates image concepts together with modification instructions into unified text queries suitable for text-only CLIP backbones.
- High Data Efficiency: GRAPE does not require the LLM to learn new external world knowledge—only how to format queries for CLIP. Consequently, training with just 10k samples captured 88%–94% of the performance achieved using full 100k-sample training splits.
Key Results
GRAPE was tested across multiple frozen CLIP backbones (ViT-B/32, ViT-B/16, and ViT-L/14):
- Consistent Retrieval Gains: On average across all five benchmarks, GRAPE outperformed frozen zero-shot LLM rewriting by +4.5% to +4.7% in R@1 and +4.9% in R@10 across backbones.
- Performance Highlights (ViT-B/16 Backbone):
- Flickr30k-CN: R@1 reached 58.0% (vs. 52.1% for frozen LLM).
- Wikipedia (Long Text): R@1 increased from 34.3% (frozen LLM) to 43.7% with GRAPE.
- CIRR (Multimodal): R@1 rose from 30.7% to 35.7% (R@10 improved from 73.3% to 77.8%).
- Scalability: Gains persisted as the underlying retriever scaled up; ViT-L/14 achieved 43.4% average R@1 with GRAPE compared to 38.9% with frozen LLM rewrites.
Limitations and Future Work
- Inference Latency: Running an LLM (or VLM) rewrite step prior to querying the vector database introduces computational overhead and query latency, posing hurdles for strict sub-millisecond production environments.
- Retriever Upper Bound: Because the underlying retriever remains frozen, GRAPE cannot overcome fundamental representational blind spots or capacity limits inherent to the CLIP encoder.
- LLM Knowledge Dependencies: GRAPE relies on the LLM's internal priors. Per-language evaluations on XM3600 revealed that retrieval gains taper off significantly for low-resource languages where the base LLM lacks strong foundational representations.
Summary updated