Modern search systems often close the gap between how users phrase a question and how documents phrase the answer by running a loop. A language model writes background text that expands the query. A retriever fetches documents using that text. The model then reads those documents and rewrites the expansion, and the cycle repeats. These loops help a great deal on queries whose answers share few words with the question, a situation that is common in practice. Yet every system we know of runs them for a fixed number of rounds, and the loop has two failure modes a fixed count cannot see. It may keep churning after it has already found the right documents. Worse, it may settle into a stable but wrong state, where the model keeps repeating vocabulary it invented and the retriever keeps returning the documents that match those invented words.
We propose a per-query stopping rule for query–document loops that watches two things about the corpus rather than the text the model generates. The first is whether the ranked list of retrieved documents has stopped changing. The second is whether the current expansion is actually supported by a passage the retriever found. When both hold, the loop stops and its ranking is trusted. When the ranking is stable but unsupported, or never stabilizes, the system abandons the expansion and falls back to the original query. In two sentences: stop a search loop when its results stop moving and its new vocabulary is backed by real documents. If the results stop moving without that backing, treat the loop as stuck in a self-reinforcing hallucination and revert to plain keyword search.
Core Claim
The vocabulary mismatch problem (often called the lexical gap) is the situation where a query and its relevant documents describe the same thing in different words. It is not marginal. WikiQA, which built an answer-selection benchmark from real Bing queries, found that 20.3% of correct answer sentences share no content words with the question.
The strongest loop-based answer to the gap is InteR. It alternates language-model knowledge generation with retrieval and reaches 50.0 MAP on TREC DL'19 with GPT-3.5, against 30.1 for BM25 and 41.8 for HyDE. The authors report that two rounds is a "sweet spot" and that further rounds add noise. That result is an average over queries. It tells us nothing about which queries were harmed at round two or helped at round three.
A second paper explains why the harm matters. Hypothetical Documents or Knowledge Leakage? checks LLM query expansions against gold evidence with an entailment model. It shows that expansion gains come mostly from claims the model had effectively memorized. On FEVER with GPT-4o-mini, "unmatched" claims reach Recall@5 of 23.8, below plain BM25 at 31.0. Read together, the two papers motivate our claim. A loop that is not grounded in the corpus can converge confidently to a false attractor, so the right stopping signal must come from the retrieval side and must check grounding, not just stability.
Concretely, let $R_t$ be the top-$k$ ranking produced at round $t$. Let $e_t$ be the expansion written at round $t$. Let $q_0$ be the original query. We measure ranking stability as
$$s_t = \mathrm{RBO}_p(R_{t-1}, R_t),$$
where $\mathrm{RBO}_p$ is rank-biased overlap, a top-weighted agreement score between two ranked lists in $[0,1]$ with persistence parameter $p$. We measure support as
$$c_t = \max_{d \in R_t^{(m)}} \mathrm{NLI}(d \Rightarrow e_t),$$
the highest entailment probability that any of the top $m$ retrieved passages $d$ gives to the expansion. This reuses the entailment-matching idea of the leakage paper, but applies it online against retrieved passages rather than offline against gold ones.
The policy has three outcomes. If $s_t \ge \tau_s$ and $c_t \ge \tau_c$, the loop stops and returns $R_t$. If $s_t \ge \tau_s$ but $c_t < \tau_c$, it has found a false attractor and returns BM25 on $q_0$. If $t$ reaches the budget $T$ without stabilizing, it also falls back. The new element is the third outcome together with the support test. Existing convergence-based stopping in reasoning, such as answer consistency in Answer Convergence as a Signal for Early Stopping in Reasoning and embedding-similarity stopping in Entrospect, treats stability as success. Both papers acknowledge that this is risky when the model is overconfident, and the leakage result is exactly the setting where it fails.
Research Program
The work has three parts. First, we reimplement an InteR-style loop with an open language model (Llama 3.1 8B, plus a larger open model for scale), a BM25 sparse retriever, and a budget of $T=5$ rounds. We log every round's expansion and top-100 ranking for every query. These logs let us compute, offline and without extra model calls, the oracle per-query best round $M^*$. Every stopping policy is then evaluated by replay.
Second, using the matched/unmatched labeling procedure from the leakage paper on FEVER, SciFact and AVeriTeC, we characterize how trajectories behave. Our hypothesis is that matched claims stabilize early with high support. Unmatched claims should split into two groups: oscillating trajectories, and stable-but-unsupported ones. The second group is the false attractor, and if it does not exist in meaningful numbers, the support test has nothing to do.
Third, we build the policy. We tune $\tau_s$ and $\tau_c$ on one TREC DL year and test on the other. The BEIR datasets are run zero-shot with thresholds carried over. As a cheaper variant of $c_t$ for latency-sensitive use, we replace entailment with best-passage term coverage, defined as the IDF-weighted fraction of the expansion's new terms (those absent from $q_0$) that occur together in a single retrieved passage. The intuition is that a hallucinated expansion strings together terms that BM25 can match in scattered documents but that rarely co-occur in any real one. Finally, we test whether a small logistic classifier on trajectory features ($s_t$, $c_t$, their changes, and score margins), trained with self-supervised labels from $M^*$, beats the two-threshold rule. This mirrors Learn-to-Stop, but on corpus-side features instead of model activations.
Baselines
The naive reference points are BM25 on the original query and the round-one expansion, which is HyDE-like single-shot expansion. The leakage paper reports BM25 at Recall@5 of 31.0 on FEVER, and unmatched expansion at 23.8. Our policy must not fall below BM25 on the unmatched subset; that is the minimum bar for the fallback.
The strongest existing loop is InteR at a fixed two rounds. InteR reports 50.0 MAP on DL'19 with GPT-3.5 and gains of more than 10 nDCG@10 points on TREC-COVID. Because we use a different language model, we report our own reproduction of InteR at $M=2$ under identical retriever, prompts and budget. Beating it means matching or exceeding its nDCG@10 and MAP while using fewer language-model calls on average, and having a lower rate of queries scored below BM25.
The strongest single-shot competitor is Aligned Query Expansion. It trains a rewriter with rejection-sampling fine-tuning and preference optimization on BM25 ranks, and reports 30.8% Top-1 on Natural Questions versus 22.1% unexpanded. We train it on the same base model with the same compute and evaluate it on our benchmarks. Because a single trained pass and a loop are not mutually exclusive, we also run AQE as the generator inside our loop.
To match a fixed compute budget, we include fixed-$M$ runs for every $M$ from 1 to 5, and the per-query oracle $M^*$ as an upper bound. Three ablations isolate our ingredients. The first is stability-only stopping, using $s_t$ with no support test or fallback. The second is text-side stopping, using cosine similarity above 0.9 between consecutive expansions, the rule Entrospect uses. The third is reciprocal-rank fusion of BM25 and fixed-$M=2$ InteR, which captures a "hedge everything" alternative to a targeted fallback. All methods share the same retriever, generator, prompts and threshold-tuning split, and report both effectiveness and model calls per query.
Why not just…?
Why not just keep the fixed two rounds? InteR's own finding that more rounds hurt on average implies the best round varies by query. HALT, which stops multi-hop search agents once a small verifier finds that fixed "hop claims" are covered, cut search loops by 45% on HotpotQA with little accuracy loss. That shows fixed budgets waste rounds in retrieval settings. Whether that headroom exists in single-query expansion loops is an empirical question. The gap between fixed $M=2$ and the oracle $M^*$ is our first decisive check, and if it is small we will say the project should stop there.
Why not just stop when the rankings stop changing? This is the most serious threat, because rank stability alone may capture most of the efficiency gain. Our argument is that a hallucinated expansion, repeated each round, produces a perfectly stable ranking. Under that argument, stability alone should be worst exactly on unmatched claims. The stability-only ablation, broken down by matched versus unmatched subsets, settles the question. If it equals the full policy on unmatched claims, the support test is unnecessary and we will report that.
Why not just stop when consecutive expansions look alike, as Entrospect does? Text similarity measures whether the model has stopped rewording itself, not whether the corpus agrees with it. The open problems noted for this approach also include insensitivity to small factual changes such as a swapped entity or number, which is exactly what separates a correct bridging expansion from a near-miss. The text-side ablation tests this directly.
Why not just train a better single-shot rewriter like AQE or RaFe? RaFe trains small rewriters from reranker feedback and reports 2–3% downstream QA gains. Aligned rewriters are cheaper per query, and on matched queries they may be enough. They have no recourse, however, when their one guess is wrong, and AQE's alignment signal comes from in-domain gold ranks. We test them out of domain on BEIR, and we test them inside our loop. If AQE alone matches our policy on unmatched claims, iteration is not worth its cost there.
Why not just fuse BM25 into every result? Fusion hedges uniformly. It dilutes good expansions on matched queries while only partly rescuing bad ones. If fusion matches our targeted fallback on both subsets, our contribution reduces to the stopping half of the rule, and the fusion baseline is there to reveal that.
Proposed Experiments
We evaluate on TREC DL'19 and DL'20 (nDCG@10, MAP), on BEIR TREC-COVID, SciFact and FEVER (nDCG@10, Recall@5 and Recall@100), and on AVeriTeC for fact verification, each split into matched and unmatched subsets using the leakage paper's entailment procedure. We also use the zero-overlap subset of WikiQA to test whether the policy correctly continues the loop where bridging is genuinely needed. On every benchmark we report average language-model calls and wall-clock time per query, along with a harm rate: the fraction of queries scored below BM25 on the original query.
Three results decide the proposal. The first is the oracle headroom between fixed $M=2$ and per-query $M^*$, which tells us whether adaptive stopping can matter at all. The second is the frequency of stable-but-unsupported trajectories among unmatched claims, and how well $c_t$ separates them from stable-and-supported ones (measured as AUROC against per-query improvement over BM25). This tests whether false attractors exist and can be detected. The third is the full policy against InteR at $M=2$ and against the stability-only ablation, compared at equal or lower average calls. The success criterion is that the policy keeps InteR's effectiveness on matched queries, stays at or above BM25 on unmatched ones, and uses fewer rounds overall.
References
- WikiQA: A Challenge Dataset for Open-Domain Question Answering
- Synergistic Interplay between Search and Large Language Models for Information Retrieval
- Hypothetical Documents or Knowledge Leakage? Rethinking LLM-based Query Expansion
- Answer Convergence as a Signal for Early Stopping in Reasoning
- Entrospect: Information-Theoretic Self-Reflection Elicits Better Response Refinement of Small Language Models
- Aligned Query Expansion: Efficient Query Expansion for Information Retrieval through LLM Alignment
- HALT: Verification-Aware Stopping for Retrieval-Augmented Search Agents
- RaFe: Ranking Feedback Improves Query Rewriting for RAG