Reinforcement learning (RL) has become a strong way to teach a language model to search. The model rewrites a user's query, sends it to a search engine, and is rewarded by how many relevant documents come back. These agents need a query paired with its correct document for every training episode, and labeled real queries are scarce. Practitioners therefore build a "simulator": an LLM reads a document and writes a question it answers. The trouble is that LLM-written questions are clean, grammatical, and reuse the document's own words, while real users type short, vague, misspelled, keyword-style queries. Robotics has seen this before. Models trained in clean simulators exploit the cleanliness and then break on real sensors.
This note proposes to measure that gap for RL search agents and then close it using only unlabeled real queries, which most search systems log in large numbers. The method trains a second small policy, a "query degrader", that edits synthetic queries until a classifier can no longer tell them apart from real logged queries. The edits must keep the query answerable by its original document, so the label stays valid. The search agent is then trained by RL on these degraded queries. The bet is that the degrader gets realistic queries and correct labels together, while each alternative sacrifices one of the two.
Core Claim
The claim has two parts, and each can fail on its own. The first is empirical. An RL query-rewriting agent trained on LLM-synthesized queries loses a measurable share of its in-simulator gains when evaluated on real human queries over the same corpus. That loss is a query-distribution gap, separate from the corpus-domain gap that benchmarks like BEIR measure. The second is methodological. A learned, label-preserving degradation of synthetic queries closes more of that gap than three cheaper options: random noise, few-shot style prompting, or pseudo-labeling the real logs.
The closest work is Reinforced IR. It adapts retrievers to new domains by co-training an LLM generator and a dense retriever on synthetic data. It treats query synthesis as a fixed prompting step and itself notes that the synthetic queries can be noisy and unrepresentative. It never models the gap between synthetic and real queries. Opt-Miner likewise trains information-seeking agents on synthesized data without quantifying the gap.
From the simulation side, the method comes from Adversarially Masking Synthetic to Mimic Real. That paper learns discrete drop masks that corrupt clean synthetic LiDAR scans until a discriminator cannot separate them from real scans. This removes the "clean versus noisy" shortcut and raises SemKITTI segmentation accuracy (mIoU) from 15.2% to 22.8%. Its 55.5% supervised result on real data also warns that such methods close only part of the gap. We carry the idea from sensor dropouts to text edits. We add an answerability constraint, which LiDAR masking never needed, because a corrupted query can silently stop matching its label.
Research Program
The search agent follows DeepRetrieval. A 3B model writes a short reasoning step and then a rewritten query. It is trained with PPO against a frozen BM25 index, and the reward is recall or NDCG. We fix BM25 deliberately. It avoids the cost of re-indexing the corpus whenever retriever weights change, a cost that R3 needed a special architecture to avoid. It also means any change in agent behavior comes from the training queries, not from a moving index.
The simulator is an LLM prompted to write a query for each passage. The gold passage is the label.
The degrader is a small language model that applies a short sequence of edits to a synthetic query. Edits can drop words, convert a question into keywords, abbreviate, misspell, swap a word for a vaguer synonym, or truncate. We train it with GRPO, a policy-gradient method that scores each sample against others from the same prompt. Its reward combines two terms. The first is the log-odds that a realism classifier assigns "real". This classifier is trained on unlabeled logged queries versus degraded synthetic queries and is refreshed during training, as in adversarial training. The second term is an answerability gate: the degraded query must still give its gold passage a high score under a frozen cross-encoder relevance model, and must still retrieve it within the top 100 under BM25. If either test fails, the reward is zero.
To keep the classifier from rewarding cheap tricks, such as always lowercasing or always cutting to two words, we monitor length and edit-type histograms against the real logs. If the classifier keys on a single surface statistic, we add matched augmentations of that statistic to its training data. The agent is then trained on a mixture of degraded and original synthetic queries.
Two analyses accompany the method. The first sweeps degraders of increasing realism, measured as the realism classifier's held-out AUC, and plots the agent's real-query performance against realism. If realism drives transfer, that curve should be monotone. The second addresses a confound DeepRetrieval flags: an agent can write answer facts from its own memory into the query. Gains on vague real queries could come from that rather than from better search. We therefore measure how often answer strings appear in the agent's rewrites, and we rerun the evaluation with rewrites that contain answer strings filtered out.
Baselines
The naive reference points come first. One is BM25 on the raw real query, which BEIR found to be a surprisingly strong zero-shot baseline. The other is the same 3B agent trained by RL on undegraded synthetic queries. That second model is the "trained only in the clean simulator" condition, and the gap is defined relative to it.
The ceiling is the same agent trained by RL on real labeled queries, using MS MARCO's real Bing training queries with their labels. It plays the role the supervised real-data result plays in the LiDAR work. We report results as the fraction of the gap closed:
$$G = \frac{P_{\text{method}} - P_{\text{syn}}}{P_{\text{real}} - P_{\text{syn}}},$$
where $P$ is NDCG@10 on held-out real queries. $P_{\text{syn}}$ comes from the agent trained on clean synthetic queries, and $P_{\text{real}}$ from the agent trained on real labeled queries.
The strongest existing method is DeepRetrieval itself, re-run in our setup. In its own paper it reports 65.07% recall on LEADS publication search, against a previous best of 24.68%. For its MS MARCO accuracy, the file reports only a 34x speedup over dense retrieval at "competitive performance". No reported result was found on Lacuna for DeepRetrieval's NDCG on real MS MARCO dev queries, so we reproduce it.
For cross-domain context we also report Reinforced IR, which reports average nDCG@10 rising from 35.4% to 52.3% with Contriever on eight low-resource BEIR datasets. It uses a dense retriever rather than BM25, so we treat it as a reference number, not a controlled comparison.
Three further baselines represent the cheaper alternatives in the next section. The first applies random edits drawn from the same edit vocabulary, matched to the degrader's average edit count and with the same answerability gate. The second prompts the query-writing LLM with sampled real logged queries as style examples. The third pseudo-labels the real logged queries: each real query gets the passage a strong cross-encoder ranks highest, and the agent is trained by RL on these labels.
Every condition uses the same passages, the same number of RL episodes, the same agent initialization, and the same tuning budget, which is a grid of a fixed size over learning rate and KL coefficient. Every condition also uses the same pool of unlabeled real queries wherever it uses real queries at all.
The proposal counts as beating the field only if two things hold. It must reach $G$ meaningfully above all three cheap baselines, with the margin larger than the seed-to-seed spread across three seeds. And it must at least match pseudo-labeling on real queries.
The key ablation removes the realism classifier and keeps the gate. Comparing it with the full method isolates the adversarial ingredient. A second ablation removes the gate and keeps the classifier. It tests whether answerability matters, or whether realism alone suffices even with corrupted labels.
Why not just…?
Why not just add random typos and keyword-truncation to the synthetic queries? This is ordinary domain randomization, and it may capture much of the effect. The LiDAR evidence argues against it: learned masks beat both no adaptation and the CoSMix baseline (20.0% mIoU) because real noise is structured, not uniform (Adversarially Masking Synthetic to Mimic Real). Real queries are vague in specific places. Users drop the rare discriminative terms, not random words. The matched random-edit baseline settles this directly. If it reaches the full method's $G$, we drop the adversarial machinery and report that realism of edit rates, not edit placement, is what matters.
Why not just show the query-writing LLM some real queries as examples? This is the most serious threat, since it is nearly free. We expect it to fall short for two reasons. LLMs tend to drift back toward fluent text even when prompted for terse queries. And making a query vague in the realistic way requires knowing which words the document makes indispensable, which a one-pass prompt does not see. The answerability gate gives the degrader exactly that feedback. Still, this is an expectation. If style prompting matches the degrader on $G$ and on realism AUC, the honest conclusion is that the gap is real but cheap to close, and the paper becomes a measurement study.
Why not just pseudo-label the real query logs and train on them directly? This uses real queries, so it has no query-distribution gap. Its cost is wrong labels: a vague real query is exactly where a cross-encoder's top passage is least reliable. RL rewards computed against wrong passages could teach the agent to chase the labeler's errors. Pseudo-labeling with a fixed generator also underperformed closed-loop adaptation in Reinforced IR, though that comparison was about corpus domains, not query style. We will measure pseudo-label precision on MS MARCO, where gold labels exist for the real queries, and plot agent performance against that precision. If pseudo-labeling wins, it shows that real query distribution matters more than clean labels for RL search agents, and that finding is also worth reporting.
Why not just prompt a frontier model to rewrite queries? DeepRetrieval reports that its RL-trained 3B agent beats GPT-4o and Claude-3.5 on the BIRD text-to-SQL benchmark and on SQuAD evidence retrieval. Search-R1 reports 20–24% relative gains over prompted retrieval baselines. We include a prompted frontier rewriter on the real test queries for completeness. This proposal is about how to train a small agent, not about whether training beats prompting.
Proposed Experiments
The primary testbed is MS MARCO passage ranking. It is unusually well suited because one corpus has both real Bing queries with labels and passages from which synthetic queries can be generated. This separates the query gap from the corpus gap. We train on synthetic queries for a training subset of passages. The unlabeled real logs are the real training queries with their labels removed. We evaluate NDCG@10 and Recall@100 on the real dev queries.
The first decisive check is whether $P_{\text{real}} - P_{\text{syn}}$ is substantially larger than the variation across seeds. If training on clean synthetic queries already transfers fully, the problem does not exist and the project stops there.
For transfer to other domains, we repeat the pipeline on the BEIR datasets DeepRetrieval used whose queries are human-written: NFCorpus, SciFact, FEVER, and HotpotQA. For each, we generate synthetic queries from the corpus and use the real training queries, unlabeled, as logs. On LEADS publication search we test whether degradation preserves DeepRetrieval's reported recall. A drop there would mean the degrader hurts domains where queries are already precise, which tells us realism should be matched per domain rather than imposed uniformly.
The final checks are the realism sweep, the two ablations, and the answer-injection audit. Together they decide whether any improvement comes from realistic queries, from valid labels, or from the agent leaking answers into its queries.
References
- BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models
- Reinforced IR: A Self-Boosting Framework For Domain-Adapted Information Retrieval
- Opt-Miner: Empowering Information-Seeking Agent with Tree-Guided Data Synthesis for Optimization Modeling
- Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation
- DeepRetrieval: Hacking Real Search Engines and Retrievers with Large Language Models via Reinforcement Learning
- Optimizing Retrieval for RAG via Reinforcement Learning
- Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning