GRAPE: Let GPRO Supervise Query Rewriting by Ranking for Retrieval

Zhaohua Zhang, Jianhuan Zhuo, Muxi Chen, Chenchen Zhao, Wenyu Jiang, Tianwen Jiang, Mingyang Chen, Yutang, Jihong Zhang, Qiuyong Xiao, Zhixun SuSubmission to ICLR 2026 · Withdrawn

TL;DR

Retraining large-scale vision-language retrievers like CLIP when query distributions shift (due to language, length, or multimodal inputs) is prohibitively expensive because it requires re-indexing billions of database items. GRAPE (Grouped Ranking-Aware Policy Optimization Enhancement) keeps the retriever frozen and trains an LLM query rewriter using Group Relative Policy Optimization (GRPO) guided by target item rankings. By substituting naive cosine similarity rewards with a corpus-relative ranking reward, GRAPE eliminates the "score inflation" trap and delivers an average +4.9% boost in Recall@10 across diverse retrieval benchmarks.

What Problem Does This Paper Address?

CLIP models underpin many real-world, billion-scale visual retrieval engines. However, CLIP's zero-shot retrieval accuracy degrades when user queries deviate from its original training distribution—such as queries in non-English languages, long-form paragraphs, or multimodal queries (image + text instructions).

While fine-tuning the retriever on new domains can improve performance, it modifies the latent embedding space. This invalidates pre-computed vector databases, forcing costly re-embedding and system redeployment. A common alternative is using a Large Language Model (LLM) at query time to rewrite downstream queries into retriever-friendly text. However, zero-shot LLMs lack awareness of CLIP’s specific embedding behavior. Furthermore, naively fine-tuning the LLM using CLIP similarity scores causes score inflation: the LLM learns to generate generic, non-discriminative buzzwords (e.g., "real-world environment," "atmosphere") that inflate similarity scores across all corpus items while degrading true retrieval rankings.

Methods

GRAPE introduces an online reinforcement learning framework that fine-tunes an LLM query rewriter while keeping the downstream retriever and database embeddings completely frozen:

  1. Constrained Query Generation: Given a query $q$, the LLM policy generates $K$ candidate rewrites formatted with <think>...</think> reasoning chains and <answer>...</answer> outputs.
  2. Frozen Retrieval Evaluation: Each valid rewrite is embedded by the frozen CLIP text encoder and scored against the candidate image pool to determine the rank $r_k \in \{1, \dots, N\}$ of the target ground-truth item.
  3. Dual Reward Structure:
  4. Format Reward ($R^f$): Rewards outputs that adhere to format tags ($+1$) and penalizes malformed responses ($-1$, skipping retrieval).
  5. Ranking-Based Reward ($R^r$): Linearly scales the rank position: $$R_k^r = 1 - \frac{2(r_k - 1)}{N - 1}$$
  6. Group Relative Policy Optimization (GRPO): Advantage estimation is computed by normalizing total rewards ($R_k = R_k^f + R_k^r$) across the $K$ candidates for that query, regularized by a KL divergence penalty against the reference model. Because GRPO's group advantage normalization is affine-invariant, any linear scaling of the ranking reward reduces directly to the relative ranking difference, guiding optimization toward true visual discrimination rather than raw similarity score hacking.

For text-only benchmarks, the authors use Qwen2.5-3B-Instruct; for multimodal tasks, they employ Qwen2-VL-7B.

Datasets

The authors evaluated GRAPE across five benchmarks addressing three distinct distribution shifts:

  • Multilingual Shifts:
  • Flickr30k-CN (Chinese image-text retrieval)
  • CVLUE (Chinese vision-language understanding)
  • XM3600 (massively multilingual benchmark spanning 36 languages)
  • Length Shifts:
  • Wikipedia (English text-to-image retrieval using long, descriptive paragraphs)
  • Modality Shifts:
  • CIRR (Composed Image Retrieval on Real-life images; query consists of a reference image plus modification text)

No new datasets were introduced.

Key Insights and Ideas

  • The Score Inflation Trap: Optimizing query rewrites directly against CLIP cosine similarity rewards backfires. The LLM exploits high-frequency semantic associations, peppering rewrites with generic descriptors that elevate similarity across both targets and distractors alike, causing Recall@1 to steadily drop during training.
  • Affine Invariance of Relative Advantages: The authors prove that normalizing rewards within a group in GRPO renders affine-mapped ranking rewards invariant to scale and offset. The driving policy signal becomes purely the relative rank order between rewrites.
  • Distinct Adaptation Mechanisms:
  • Cross-lingual: Recovers implicit semantic entities omitted during direct translation (e.g., inferring "baseball player" instead of just literal "pitching").
  • Long-text: Distills multi-hundred-word documents down to visually salient anchor words while purging conversational filler.
  • Multimodal fusion: Translates image concepts together with modification instructions into unified text queries suitable for text-only CLIP backbones.
  • High Data Efficiency: GRAPE does not require the LLM to learn new external world knowledge—only how to format queries for CLIP. Consequently, training with just 10k samples captured 88%–94% of the performance achieved using full 100k-sample training splits.

Key Results

GRAPE was tested across multiple frozen CLIP backbones (ViT-B/32, ViT-B/16, and ViT-L/14):

  • Consistent Retrieval Gains: On average across all five benchmarks, GRAPE outperformed frozen zero-shot LLM rewriting by +4.5% to +4.7% in R@1 and +4.9% in R@10 across backbones.
  • Performance Highlights (ViT-B/16 Backbone):
  • Flickr30k-CN: R@1 reached 58.0% (vs. 52.1% for frozen LLM).
  • Wikipedia (Long Text): R@1 increased from 34.3% (frozen LLM) to 43.7% with GRAPE.
  • CIRR (Multimodal): R@1 rose from 30.7% to 35.7% (R@10 improved from 73.3% to 77.8%).
  • Scalability: Gains persisted as the underlying retriever scaled up; ViT-L/14 achieved 43.4% average R@1 with GRAPE compared to 38.9% with frozen LLM rewrites.

Limitations and Future Work

  • Inference Latency: Running an LLM (or VLM) rewrite step prior to querying the vector database introduces computational overhead and query latency, posing hurdles for strict sub-millisecond production environments.
  • Retriever Upper Bound: Because the underlying retriever remains frozen, GRAPE cannot overcome fundamental representational blind spots or capacity limits inherent to the CLIP encoder.
  • LLM Knowledge Dependencies: GRAPE relies on the LLM's internal priors. Per-language evaluations on XM3600 revealed that retrieval gains taper off significantly for low-resource languages where the base LLM lacks strong foundational representations.

Summary updated

Abstract

The CLIP model has become a cornerstone of large-scale retrieval systems by aligning text and image data in a unified embedding space. Despite its simplicity and efficiency, CLIP struggles when applied to tasks whose input distributions diverge from its training corpus, such as queries with multilingual, long-form, or multimodal differences. To avoid costly retraining, existing methods mainly adopt query-rewriting strategies with large language models (LLMs), aiming to mitigate distribution gaps at the query level. However, due to the lack of supervision signals, LLMs fail to generate the optimal one that fits the training distribution. We address this challenge with GRAPE (\textbf{G}rouped \textbf{R}anking-\textbf{A}ware \textbf{P}olicy Optimization \textbf{E}nhancement), a plug-and-play enhancement approach that incorporates ranking signals into retrieval-guided query rewriting with LLMs. Intuitively, GRAPE proposes to leverage GRPO to bridge distributional differences—including length, multilingual, and modality shifts—by transforming queries into forms better aligned with the retriever’s training distribution. However, our preliminary experiment finds that naively finetuning LLM with similarity scores can lead to score inflation, where nearly all candidates are assigned unexpectedly high scores regardless of their true relevance. To address score inflation, we propose a corpus-relative ranking-based reward, which explicitly aligns optimization with ranking metrics while suppressing spurious score inflation. Extensive experiments demonstrate that GRAPE consistently improves retrieval performance under distributional shifts—including multilingual differences (Flickr30k-CN, CVLUE, XM3600), length differences (Wikipedia), and multimodal differences (CIRR)—achieving an average improvement of 4.9\% in Recall@10.
BibTeX
@misc{zhang2026grape,
  author = {Zhaohua Zhang and Jianhuan Zhuo and Muxi Chen and Chenchen Zhao and Wenyu Jiang and Tianwen Jiang and Mingyang Chen and Yutang and Jihong Zhang and Qiuyong Xiao and Zhixun Su},
  title = {{GRAPE: Let GPRO Supervise Query Rewriting by Ranking for Retrieval}},
  year = {2026},
  url = {https://openreview.net/forum?id=nqNOcGsZy2},
  howpublished = {OpenReview},
  note = {Submitted to ICLR 2026},
}

Loading related content...

Figure