Particle-Filtered Query Rewriting for the Lexical Gap

Search engines often fail when a user's words differ from the words used in the documents that answer them. Someone types "broken pipe" while the manual says "conduit fracture"; a scientific claim is refuted by a methods paragraph that shares none of its vocabulary. A strong current fix lets a large language model rewrite the query, retrieve with the rewrite, read the retrieved documents, and rewrite again. This loop works for one or two rounds and then gets worse, because noisy documents from one round contaminate the next rewrite and the query drifts away from what the user meant. We propose to run this loop as a particle filter. A particle filter, or sequential Monte Carlo, is a method that tracks many weighted hypotheses at once and repeatedly discards weak ones while copying strong ones. Here each particle is one chain of query rewrites, scored by how relevant its retrieved documents are to the original query. Only evidence from high-scoring chains is fed back into the next round of rewriting.

The idea in two sentences: iterative query expansion degrades because it feeds too much low-quality evidence back into itself, which is a data-quality problem inside the feedback loop and not a lack of more iterations. Resampling a population of rewrite chains against a fixed relevance target keeps the evidence pool small and clean. That should let additional rounds of rewriting help instead of hurt, at the same compute budget.

Core Claim

The closest existing work is InteR (Synergistic Interplay between Search and Large Language Models for Information Retrieval). InteR alternates between a lexical retriever and a language model that writes background passages from the retrieved documents. It reaches 50.0 mean average precision (MAP) on TREC Deep Learning 2019 with GPT-3.5, against 30.1 for BM25 and 41.8 for HyDE. Its authors also report that going beyond two iterations adds noise. InteR keeps a single chain, passes forward whatever the retriever returned, and has no mechanism to recover from a bad round. RaFe (RaFe: Ranking Feedback Improves Query Rewriting for RAG) trains a small rewriter from cross-encoder reranker feedback. It shows that a reranker is a cheap and usable relevance signal, roughly 100x faster to obtain than LLM judgments, but its rewriter commits to one rewrite in one pass. MINDER (Multiview Identifiers Enhanced Generative Retrieval) bridges the gap offline with synthetic pseudo-queries and does not adapt at query time.

Our claim is specific. Suppose rewrite chains are treated as particles with three properties: they are weighted by reranker relevance to the fixed original query, they are resampled, and each next rewrite is conditioned only on evidence from surviving particles. Then, at matched language-model and retrieval budgets, three things should hold. Retrieval quality should keep rising past the two-iteration point where InteR plateaus. The gains should concentrate on queries with little word overlap with their relevant documents. And the method should beat both a single chain and the same number of independent parallel chains.

Method

Let $q$ be the original query and let $z_{1:t}$ be a chain of $t$ rewrites, each produced by a language model prompted with $q$, the previous rewrite, and a small evidence set. Let $D(z_t)$ be the top-$k$ documents BM25 retrieves for rewrite $z_t$. Let $r(q, D)$ be the mean cross-encoder score of those documents against the original query $q$, not against the rewrite. The particle filter targets the distribution

$$\pi_t(z_{1:t}) \propto p_{\text{LM}}(z_{1:t} \mid q)\,\exp\big(\lambda\, r(q, D(z_t))\big),$$

where $p_{\text{LM}}$ is the language model's own sampling distribution and $\lambda$ is a temperature that controls how sharply relevance is rewarded. Because new rewrites are proposed from $p_{\text{LM}}$, each particle's incremental weight is simply

$$w_t^{(i)} = \exp\big(\lambda\,[r(q, D(z_t^{(i)})) - r(q, D(z_{t-1}^{(i)}))]\big).$$

A particle's weight therefore rises only if its newest rewrite retrieved documents that look more relevant to what the user actually asked. Anchoring the score to $q$ is the guard against drift. A rewrite that wanders into a neighboring topic may retrieve documents that fit the rewrite well, but they will score poorly against $q$.

The loop has four steps, following the propose–weight–resample structure of standard sequential Monte Carlo as used for streaming text in Context as Filtering.

  1. Each of $N$ particles proposes a new rewrite.
  2. Each new rewrite retrieves its documents and is scored.
  3. When the weights become too uneven, particles are resampled.
  4. The evidence passed into the next round's prompt is built only from the top documents of the resampled particles.

That last step is where the data-quality direction enters. It plays the role curation plays in fine-tuning, where SHED and An Experimental Design Framework for Label-Efficient Supervised Finetuning of Large Language Models show that a small, well-chosen subset can match or beat the full pool. We apply the same principle inside a single query's feedback loop rather than to a training set.

We borrow one more element from Critic Sequential Monte Carlo, which scores many candidate actions with a cheap learned critic before running expensive simulation. Here, a small bi-encoder scores several candidate rewrites per particle against $q$ before any retrieval or cross-encoder call. This lets more candidates be considered under a fixed retrieval budget. The Critic SMC paper also warns that a wrong critic can mislead the search, so we keep this pre-screen as a separable component and test it on its own. Final output is the union of documents from surviving particles, reranked by the same cross-encoder used by every baseline.

Research Program

The work has three parts.

The first part is building the filter on an open instruction-tuned model, so that every compared method uses the same generator. We re-implement InteR on this same model, since its 50.0 MAP used GPT-3.5 and is not directly comparable to an open model.

The second part is the drift analysis. For each method we plot retrieval quality against the number of rewrite rounds, from one to five. We also measure semantic drift directly, as the embedding distance between the current rewrite and $q$. The prediction is that InteR's curve peaks near two rounds while the filter's curve keeps rising or flattens. We will further check whether the size and reranker score of the evidence fed back into the prompt explains the difference.

The third part is a lexical-gap stratification. Following the observation in WikiQA that many correct answers share no content words with the question, we bucket test queries by content-word overlap with their judged-relevant documents. We then report results per bucket.

Baselines

The naive reference point is BM25 on the unmodified query. InteR reports 30.1 MAP for it on TREC DL'19. We will also report BM25 followed by the shared cross-encoder reranker, because every other method gets that final reranking and the comparison should isolate retrieval-side gains. docT5query document expansion is the cheap static alternative. BEIR reports that it beats BM25 on 11 of 18 datasets. Like BM25, it needs no query-time language model calls.

The strongest existing method is InteR, which reports 50.0 MAP on TREC DL'19 and more than 10 points of nDCG@10 over baselines on TREC-COVID. We will run it with our open model at its best iteration count and at every iteration count we test. It is the strongest competitor because it already uses evidence feedback from retrieval to cross the vocabulary gap, so any gain we show must come from the population and the curation, not from feedback itself. A one-shot RaFe-style rewriter trained with reranker preference feedback represents the deterministic single-rewrite approach. RaFe reports 2–3% downstream QA gains on NQ-based evaluation but no per-dataset retrieval breakdown in our file. HyDE is reported at 41.8 MAP on TREC DL'19 in the InteR paper.

Fairness is enforced by fixing three totals per query across all methods: the number of generated tokens, the number of retrieval calls, and the number of cross-encoder document scorings. Hyperparameters ($\lambda$, $N$, $k$, number of rounds) are tuned on TREC DL'19 only, with an equal tuning budget for InteR's iteration count and passage length. We then report TREC DL'20 and BEIR as held-out sets.

Three ablations isolate the key ingredients:

  1. Removing resampling, which turns the filter into $N$ independent InteR chains with weighted pooling at the end.
  2. Scoring rewrites against themselves instead of against $q$.
  3. Feeding all retrieved evidence forward instead of only the surviving particles' top documents.

Beating InteR means a statistically significant MAP and nDCG@10 gain over the same-model InteR on TREC DL'20 and on the low-overlap bucket of the BEIR subsets, at matched budget.

Why not just…?

Why not just sample many rewrites in parallel and rerank the union? This is the most serious threat. If the reranker does all the work, independent sampling plus reranking should match the filter. The no-resampling ablation is exactly this baseline at equal budget. If it ties the full method, we will conclude that the gains come from the original-query anchoring and evidence filtering rather than from sequential resampling, and we will reframe the contribution around those components. Our expectation that resampling matters rests on the drift result in InteR: independent chains each pass through their own noisy second and third rounds, while resampling redirects budget away from chains whose evidence has already gone bad.

Why not just use beam search over rewrites with reranker scores? Beam search keeps the top-$N$ chains deterministically, and it may match the filter on queries with one clear meaning. We include it as a baseline at the same width. The filter's stochastic resampling should matter for ambiguous queries, where the top-$N$ beams can collapse onto one interpretation. We test this by reporting results separately on queries where the relevant documents are spread across several judged subtopics. If beam search ties there too, the simpler method wins and we will say so.

Why not just use a one-shot fine-tuned rewriter like RaFe? It is far cheaper, which is why it is a required baseline. However, a single rewrite cannot revise itself after seeing that its retrieved documents are off-topic. The InteR results show that evidence feedback adds substantially over feedback-free expansion such as HyDE. We expect the one-shot rewriter to be competitive on high-overlap queries and to lose on the zero-overlap bucket, and the per-bucket results will show whether that holds.

Why not just BM25 or docT5query? Both are fast and robust, and BEIR shows BM25 often beats dense retrievers out of domain. But lexical matching cannot retrieve a relevant document that shares no words with the query. The 30.1 versus 50.0 MAP gap on TREC DL'19 shows how much is left on the table. We report the cost–quality trade-off explicitly, so a reader can see whether the gain is worth the extra language model calls.

Proposed Experiments

Retrieval experiments use the TREC DL'19 and DL'20 human relevance judgments (MAP, nDCG@10, Recall@1000), plus TREC-COVID and SciFact from BEIR (nDCG@10). The last two are chosen because they combine specialized vocabulary with asymmetric phrasing between claims and evidence. A downstream check on Natural Questions measures answer accuracy when the top retrieved passages are given to a fixed reader model, to confirm that retrieval gains carry over to QA.

A known risk is that particle weights come from the same cross-encoder used for final reranking. The filter could then learn to please that reranker without improving true relevance, a version of the reranker-bias problem RaFe also inherits. We address this in two ways. All headline metrics use human relevance judgments. We also run a variant in which weighting uses one cross-encoder and final reranking uses a different one.

Three results would decide the question. First, does the filter's quality keep improving past two rounds while InteR's declines, at equal budget? Second, does the full method beat both independent parallel chains and beam search, especially on low-overlap and multi-subtopic queries? Third, does the advantage survive when weighting and evaluation use different rerankers? If all three hold, the result is that iterative query expansion fails because it lets too much noisy evidence back into the loop, and that a particle filter fixes this. If only the second fails, the contribution narrows to original-query anchoring plus evidence curation, which is still a useful and testable finding.