A popular, inexpensive way to teach a small model to rewrite search queries removes the human from the loop entirely. The model proposes several rewrites. Each rewrite retrieves documents, and an off-the-shelf reranker scores those documents (a reranker is a model that reads a query and a document together and outputs a relevance score). Rewrites whose documents score higher become the "preferred" side of training pairs. RaFe showed this works and is cheap: reranker scoring took 0.67 hours for 30,000 instances, where LLM-based answer checking took 78 hours. The method has a blind spot exactly where query rewriting is most needed. Rewriting matters most under vocabulary mismatch, where the user's words and the words of the answering document do not overlap. Rerankers, however, are partly lexical-overlap detectors. When they score documents against the user's original phrasing, they may systematically prefer rewrites that retrieve documents echoing the user's words. Those rewrites are the opposite of the vocabulary-bridging rewrites we want.
We propose to test this suspected bias directly and, if it holds, to remove it without using gold labels. The test asks whether reranker-derived preferences disagree with gold-evidence preferences more often as query–evidence word overlap shrinks. The fix forms preference pairs only between rewrites whose retrieved documents have matched lexical overlap with the original query. Under that matching, overlap cannot explain which rewrite wins. We train a small two-round rewriter with offline Direct Preference Optimization (DPO, which fits the policy directly to preference pairs without a separate reward model or online sampling). Its second query is conditioned on the documents from the first round, so it can borrow the corpus's own vocabulary. The intended outcome is a rewriter that closes the lexical gap on long-tail and jargon-heavy retrieval at a fraction of reinforcement-learning cost.
Core Claim
Preference signals produced by a reranker are confounded by surface word overlap between the user query and retrieved text. This confound is a concrete case of the causal confusion documented in preference-based reward learning. A Study of Causal Confusion in Preference-Based Reward Learning showed that a preference model can be 92.8% accurate on held-out pairs while the policy trained against it collapses from 97% to 1.3% task success, because the model latched onto a correlated nuisance feature. In query refinement, the nuisance feature is lexical overlap. It correlates with relevance on easy, high-overlap questions, and it anti-correlates with what the rewriter must learn on zero-overlap questions.
Yang et al. report that a substantial fraction of correct answers in open-domain QA share no content words with the query, so this regime is not marginal. Our claim is falsifiable in two steps. First, reranker preferences should agree less with gold-evidence preferences as overlap falls. Second, overlap-matched pair construction should recover most of the zero-overlap recall that a gold-label oracle achieves, at equal compute.
Relation to Prior Work
The closest work is RaFe. It supplies our training signal and our efficiency argument, but it is single-turn and treats the reranker as an infallible relevance oracle. It notes qualitatively that its rewriter learned keyword bursts, and it never asks whether that behavior is genuine improvement or score inflation.
CONQRR motivates the same worry from the reinforcement-learning side. Its rewriter, rewarded on gold-passage rank, learned long, repetitive keyword chains. It also needed about 4.5 times the training time of supervised baselines, which is why we restrict ourselves to offline preference optimization.
EVO-RAG already applies multi-head DPO to multi-step query rewriting. We therefore do not claim multi-step preference-trained rewriting as new. EVO-RAG targets the depth and redundancy of multi-hop decomposition with a seven-factor hand-tuned reward, and it does not study vocabulary mismatch or confounding in the preference labels.
Chain-of-Rewrite shows the payoff of a second, evidence-informed query on long-tail entities: PopQA Recall@5 rises from 37.4% to 59.8%. It does this with zero-shot prompting of a 13B model and no training, so it is our strongest untrained competitor.
Tree-search methods are relevant mainly as a constraint on our design. SeRTS and Think&Cite report heavy rollout cost, and SeRTS reports forgetting after training with Proximal Policy Optimization (PPO, an online reinforcement-learning algorithm). These results argue against online search-and-train loops.
Yong et al. warn that DPO's implicit reward can generalize poorly to novel prompts. That is a real threat to our out-of-domain transfer, so we test it explicitly rather than assume it away.
Research Program
The program has three parts, in order of dependence.
The diagnostic. For each training question with gold evidence, we sample several first-round rewrites, retrieve the top documents, and record two preferences per pair. The first is the reranker's preference, computed RaFe-style by scoring documents against the original question. The second is a gold preference based on gold-evidence recall. We then bucket questions by content-word overlap between the original question and its gold evidence, and plot reranker–gold agreement per bucket. We also log simple hacking signatures of the winning rewrites: the fraction of tokens copied from the user query, rewrite length, and repetition.
The fix. For any pair, we compute the BM25 similarity between the original question and each rewrite's retrieved set (BM25 is the standard word-overlap ranking function). We keep only pairs whose overlap values fall within a small caliper of each other, and label them by reranker score. Within a matched pair, a reranker preference cannot be driven by overlap with the user's words. A softer variant replaces matching with the residual of the reranker score after regressing it on that overlap, fit on unlabeled questions. Both variants need no gold labels, so they apply to deployment corpora. The gold labels are used only to measure success.
The policy. The rewriter is a roughly 2B-parameter model, comparable to the Gemma-2B refiner in Refiner-RL. It is first supervised on LLM-distilled rewrites, as in RaFe. It then runs in two rounds: round two sees the question plus the top round-one passages and emits a refined query. Each round receives its own DPO pairs from the matched construction, and round-two pairs are drawn from the policy's own round-one outputs, which reduces train–test mismatch. Total training cost (sampling, retrieval, reranking, DPO) is logged in GPU-hours, so efficiency is reported rather than asserted.
Baselines
The naive reference points are BM25 on the raw question and the supervised-only rewriter. Chain-of-Rewrite reports 34.6% accuracy and 37.4% Recall@5 on PopQA for BM25. These anchor how much any rewriting helps.
The strongest trained competitor is RaFe-style DPO with unmatched reranker preferences, in two forms: single-round as published, and applied twice in our two-round loop. Both use the same rewriter, data, reranker, sampling budget, and DPO tuning budget as our method. RaFe reports a 2 to 3% downstream QA accuracy gain over baselines across NQ, TriviaQA and HotpotQA. No reported result was found on Lacuna for RaFe on PopQA or LitSearch, so we rerun it ourselves. Beating it means a statistically reliable Recall@5 gain on the zero-overlap bucket, with no loss on the high-overlap bucket.
The strongest untrained competitor is Chain-of-Rewrite. It reports 59.8% Recall@5 and 48.0% accuracy on PopQA with Vicuna-13B. We rerun it with the same retriever and corpus, and report inference FLOPs and latency next to recall. A small trained model that matches this recall at much lower inference cost would count as a win. Exceeding the recall is not required.
On LitSearch, the strongest single-shot retriever reported is GritLM-7B at 74.8% recall@5 on specific questions (LitSearch). We place our rewriter in front of that same retriever, so the comparison isolates what query refinement adds.
An oracle baseline trains DPO on gold-evidence preferences. It is the ceiling for any label-free signal, and our target is to close most of the gap between unmatched-reranker DPO and this oracle.
Two ablations isolate the ingredients. The first is matched pairs without the second round, which tests deconfounding alone. The second is a second round with unmatched pairs, which tests iteration alone. A third ablation, random pair subsampling at the same pair count as matching, rules out the possibility that matching helps merely by discarding noisy pairs.
Why not just…?
Why not just use a strong dense retriever and skip rewriting? This is a serious threat on LitSearch, where dense models beat BM25 by nearly 25% in recall. BEIR, however, found that dense retrievers often fall below BM25 out of domain. CONQRR found that a learned rewriter raised MRR by 15% for both BM25 and a dense dual encoder, suggesting the gains are complementary. We settle this directly by putting the rewriter in front of GritLM-7B and reporting gains per overlap bucket. If the gain is near zero even on zero-overlap questions, the proposal's practical value on strong retrievers is small, and we will say so.
Why not just use gold answers or gold passages as the reward, which are cheap and unconfounded? On PopQA they are. That is exactly why we use them as the oracle and the measuring stick. The method targets corpora such as specialized catalogs, legal codes and evolving scientific literature, where no gold evidence exists at training time but a reranker does. If label-free matched preferences reach the oracle on benchmarks that do have labels, that is evidence they can stand in where labels are absent.
Why not just prompt a large model to refine iteratively, as Chain-of-Rewrite does? This works, at 59.8% Recall@5 on PopQA. It requires several 13B-scale passes per query at inference, and the authors themselves flag distillation as necessary. Our comparison reports recall per unit of inference compute. If the prompted model wins on that axis too, then the training effort is not justified.
Why not just run RaFe twice? This is the iteration-only ablation above. If it matches our method, then deconfounding contributed nothing, and the diagnostic's prediction was wrong.
Proposed Experiments and Decisive Checks
We train on the NQ and PopQA training splits against Wikipedia. We evaluate in-domain on PopQA, NQ serves as a high-overlap control, and the out-of-domain tests are LitSearch and BioASQ Task 11b. LitSearch matters for a second reason: its queries come from citation contexts and authors rather than BM25 pooling. That avoids the lexical bias in benchmark construction that BEIR's authors warn can hide vocabulary-gap failures. The primary metric is Recall@5 on the zero-content-word-overlap bucket. Secondary metrics are overall Recall@5, downstream reader accuracy, hacking signatures, and total training GPU-hours compared against a PPO run of the RaFe reward at matched sample count.
The idea fails if any of three checks fails. First, reranker–gold agreement must fall with decreasing overlap; a flat curve would show the confound does not exist. Second, matched-pair DPO must beat both unmatched DPO and random subsampling on the zero-overlap bucket. Third, the gain must survive transfer to LitSearch and BioASQ, which is where Yong et al.'s warning about implicit-reward generalization would show up.
A positive result is a cheap, label-free correction to a widely used training signal, aimed at the questions where retrieval most often fails. A negative result on the first check is also informative, because it would show that reranker preferences are safer for vocabulary bridging than the causal-confusion literature suggests.
References
- RaFe: Ranking Feedback Improves Query Rewriting for RAG
- Causal Confusion and Reward Misidentification in Preference-Based Reward Learning
- WikiQA: A Challenge Dataset for Open-Domain Question Answering
- CONQRR: Conversational Query Rewriting for Retrieval with Reinforcement Learning
- Curriculum Guided Reinforcement Learning for Efficient Multi Hop Retrieval Augmented Generation
- Chain-of-Rewrite: Aligning Question and Documents for Open-Domain Question Answering
- SeRTS: Self-Rewarding Tree Search for Biomedical Retrieval-Augmented Generation
- Think&Cite: Improving Attributed Text Generation with Self-Guided Tree Search and Progress Reward Modeling
- On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization
- Enhancing the Capability and Robustness of Large Language Models through Reinforcement Learning-Driven Query Refinement
- LitSearch: A Retrieval Benchmark for Scientific Literature Search
- BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models