Large language models can rescue search when a user's words do not match the words in the relevant documents. This failure, called vocabulary mismatch or the lexical gap, is common: a large share of correct documents in open-domain question answering share no content words with the question. The standard fix is to ask the model to reason step by step, called chain-of-thought (CoT), and then write a better search query. A single reasoning chain is brittle, though, and the usual remedy of sampling many chains and taking a majority vote breaks down here. Two good rewrites almost never match word for word, so there is nothing to count. We propose to move the vote out of word space and into the space of retrieved documents. Two rewrites "agree" if they retrieve the same documents, however differently they are phrased.
The second ingredient tells trustworthy chains from untrustworthy ones. For each sampled chain we cut the reasoning partway through, let the model finish it several more times, and check whether the new queries retrieve the same documents as the original. Chains whose retrieved set survives this regeneration get more weight when the ranked lists are merged. Chains whose results scatter get less. The claim is that stability under regeneration, measured in document space rather than in tokens, is a cheap and label-free signal of whether a rewrite has found the right concept. It should let a small number of sampled chains match or beat much larger unweighted ensembles and iterative search-and-generate loops.
Core Claim
Work on probabilistic inconsistency shows that language models do not define coherent distributions over text. CONTESTS measures how much a span's probability changes when its tokens are scored in a different order. All 18 models it tested failed on Wikitext, and autoregressive models got worse with scale. From Drift to Coherence shows that a model's answers drift when it is asked the same question repeatedly, which violates the martingale property: the expected future belief should equal the current one. For retrieval, though, token-level inconsistency is the wrong thing to measure. A query rewriter can be wildly inconsistent over word strings and still perfectly consistent over the documents those strings retrieve.
Our central claim is that the right consistency test for CoT query rewriting looks at the distribution over retrieved documents that the model's continuations induce. Concretely, if a reasoning prefix has committed to the correct underlying concept, its continuations should retrieve overlapping document sets even when they differ lexically. A prefix that has drifted should not. This is a retrieval analogue of the discrimination gap reported by Reliable Chain-of-Thought via Prefix Consistency. That paper truncates math reasoning traces, regenerates continuations, and finds that correct traces reproduce their final answer more often than wrong ones. It only works when the answer is a discrete token that can be matched exactly. Document identifiers are discrete even when queries are not, and that is what makes the extension possible.
Research Program
The method works as follows. Given a query $q$, the model samples $m$ CoT traces. Each trace reasons about what the user means and what terminology relevant documents would use, then emits a search query $q_i$. For each trace we truncate the reasoning at fraction $\tau$ of its length and regenerate $K$ continuations, which produce queries $q_{i1},\dots,q_{iK}$. We run all of them through the same retriever. The stability score $c_i$ of trace $i$ is the mean rank-biased overlap between the top-$k$ list of $q_i$ and the top-$k$ lists of its regenerated queries. Rank-biased overlap is a top-weighted similarity between two ranked lists. Retrieved lists are then merged by stability-weighted reciprocal rank fusion:
$$s(d) = \sum_{i=1}^{m} \frac{c_i}{60 + \mathrm{rank}_i(d)},$$
Here $\mathrm{rank}_i(d)$ is document $d$'s position in the list retrieved by $q_i$, and 60 is the conventional fusion constant. Setting every $c_i = 1$ recovers ordinary multi-query fusion, which is our main ablation.
The program has three parts. First, a diagnostic study tests whether the retrieval-space discrimination gap exists at all. Using relevance judgments, we compare the mean stability of traces whose query puts a relevant document in the top 10 against traces whose query does not. We break this down by how much lexical overlap the original query has with its relevant documents. We expect the gap to be largest exactly where the vocabulary gap is worst. We also check whether it holds across $\tau$ and $K$. Second, an efficiency study compares accuracy against generated tokens, since prefix regeneration costs tokens. Third, we apply the same score as a stopping rule for iterative retrieval-generation loops. InteR alternates LLM knowledge generation with retrieval and found that more than two rounds hurt because noise accumulates. We test whether halting when the stability of the next round's query drops below that of the current round removes the need for that fixed cutoff. This is a weak, test-time version of the martingale condition from From Drift to Coherence: keep iterating only while the expected retrieval outcome is not moving away from the current one.
Prior work on the lexical gap motivates this design, and some of it supplies the method. Generative expansion such as InteR and HyDE, and learned rewriters such as RaFe, show that LLM rewrites close much of the gap. RaFe trains a small rewriter with preference feedback from a cross-encoder reranker. None of these methods estimates per-rewrite reliability without an external scorer, and RaFe's rewriter inherits its reranker's biases. CoTKR uses interleaved reasoning to rewrite knowledge-graph facts into text and trains on downstream QA feedback. It works on structured subgraphs and never measures stability. Self-Consistency supplies the sample-and-aggregate idea but needs exact-match answers. Dual Supervised Learning enforces $P(x)P(y\mid x)=P(y)P(x\mid y)$ during training. We considered a duality-constrained rewriter of this kind and decided against it. It needs marginals $P(q)$ and $P(q')$ that are hard to estimate for open-domain queries. It also only checks that the rewrite preserves the user's wording, not that it reaches the right documents.
Two results constrain the proposal. From Drift to Coherence shows that a fully coherent model can still be confidently wrong. Grounding Fallacies shows that retrieval readily finds lexically overlapping "kernel of truth" passages while missing the refuting evidence. When Llama 3 was fed such passages, its acceptance of false claims rose from 0.0% to 44.4%. Stability weighting could make this worse by rewarding rewrites that reliably return the same misleading passage. We treat this as a falsification target, not a footnote.
Baselines
The naive reference points are BM25 on the raw query and a single greedy CoT rewrite fed to BM25. BM25 is the right floor because BEIR shows it often beats out-of-domain dense retrievers. On TREC DL'19, InteR reports BM25 at MAP 30.1.
The strongest existing method on our primary benchmark is InteR, which reports MAP 50.0 on TREC DL'19 with GPT-3.5, against 41.8 for HyDE (InteR). On TREC DL'20 and the BEIR subsets it reports only relative gains: more than 8 points in MAP and more than 5 in nDCG@10, and over 10 nDCG@10 points on TREC-COVID. We found no reported absolute InteR result on Lacuna for SciFact. We will quote the published 50.0 and also rerun InteR and HyDE with our own open-weight model, retriever, and corpus. Comparisons within our setup will use the rerun numbers.
The most dangerous baseline is unweighted multi-query fusion: the same $m$ CoT rewrites merged by plain reciprocal rank fusion. It isolates our key ingredient, the stability weight. Because prefix regeneration spends extra tokens, we run unweighted fusion with $m' > m$ chains so that both methods use the same token count, and we report full accuracy-versus-tokens curves.
We also include a reranker-selection baseline. A cross-encoder reranker such as the BGE reranker used by RaFe reorders the union of all retrieved documents, or picks the single best rewrite. This is the label-free signal a practitioner would reach for first. Where training budget allows, we include RaFe-style trained rewriters for downstream QA. RaFe reports 2% to 3% QA accuracy gains over supervised fine-tuning on NQ, TriviaQA, and HotpotQA (RaFe).
All methods share the generator, the retriever, the top-$k$, and a tuning budget. $\tau$, $K$, and the fusion constant are selected on TREC DL'19 only and frozen for every other dataset. To count as beating the strongest competitor, our method must exceed InteR's MAP on TREC DL'19 under our rerun. It must also beat equal-token unweighted fusion by a margin outside bootstrap confidence intervals, on TREC DL'20 and on at least four of six BEIR subsets.
Why not just…?
Why not just sample more rewrites and fuse them without weights? This is the real threat. Fusion already rewards documents that many rewrites retrieve, which is a form of agreement in document space. Our weight adds something only if a stable trace predicts quality beyond what simple popularity across traces captures. The equal-token ablation decides this directly. If unweighted fusion with more samples matches us at the same token count, the stability score adds nothing and we will report that. The diagnostic study gives an earlier warning: if the retrieval-space discrimination gap does not survive after controlling for how many traces retrieve a document, the weighting cannot help.
Why not just rerank the union of retrieved documents with a cross-encoder? A reranker is a strong, cheap source of relevance signal, and BEIR shows that BM25 combined with a cross-encoder generalizes well. It can only rerank documents that some rewrite already retrieved, though, so it cannot fix a pool dominated by drifting rewrites. It also brings its own domain blind spots, as the RaFe authors note. Stability weighting and reranking are complementary, so we will report the combination as well. If reranking alone captures all the gain, our contribution shrinks to an efficiency result about selecting good chains before the reranker runs.
Why not just cluster the generated queries or answers by embedding similarity and vote within clusters? Embedding similarity measures closeness in wording and meaning, not retrieval outcome. Two queries can be close in an embedding space and retrieve different documents across a jargon boundary. Two queries can also differ lexically and retrieve identical documents. We will test this directly by replacing rank-biased overlap with cosine similarity between query embeddings when computing $c_i$. If embedding similarity matches retrieval-overlap stability, document-space voting is not necessary.
Why not just train a cycle-consistent rewriter in the style of Dual Supervised Learning? A round-trip constraint penalizes rewrites that drift from the user's wording. Closing a lexical gap, however, requires the rewrite to drift lexically while keeping its meaning. Duality-style training also needs marginal estimates that Dual Supervised Learning itself identifies as noisy in high dimensions. We therefore leave training-time constraints to follow-up work, conditional on the test-time signal proving real.
Proposed Experiments
The first decisive check is the diagnostic one. On TREC DL'19/'20 and the BEIR subsets TREC-COVID and SciFact, with relevance judgments in hand, does the stability of traces whose rewrite retrieves a relevant document exceed that of traces whose rewrite does not? Does the gap grow as query-document lexical overlap shrinks? If the gap is near zero, the project stops there, and that negative result about where the prefix-consistency signal of Reliable Chain-of-Thought via Prefix Consistency transfers is itself worth reporting.
The second check covers ranking quality and efficiency. We report MAP and nDCG@10 on TREC DL and nDCG@10 on six BEIR subsets, against all baselines at matched token budgets. Prefix consistency reports reaching standard voting plateaus with a median of 4.6x fewer tokens on math. A comparable saving here would be a strong practical result.
The third check is downstream question answering. On NQ, TriviaQA, HotpotQA, and FreshQA as an out-of-domain test, we feed the fused top passages to a fixed reader and compare against RaFe-style rewriting.
The fourth check is the stopping rule. We run InteR-style loops to $M=4$ rounds, with and without the stability-based halt, and ask whether performance stops degrading after round two.
The fifth check is the adversarial one. On SciFact-Open, where MultiVerS falls from 72.5 F1 on the original SciFact to 52.4 F1 (SciFact-Open), and on MISCIPLUS (Grounding Fallacies), we measure whether stability weighting raises or lowers recall of refuting and limiting evidence relative to kernel-of-truth passages, and how it changes downstream false-claim acceptance. A rise in kernel-of-truth concentration would confirm that coherence is not correctness in retrieval as well. It would mark the boundary of the method: stability weighting finds consensus terminology, and a separate mechanism is needed to seek out counter-evidence.
References
- CONTESTS: a Framework for Consistency Testing of Span Probabilities in Language Models
- From Drift to Coherence: Stabilizing Beliefs in LLMs
- Reliable Chain-of-Thought via Prefix Consistency
- Synergistic Interplay between Search and Large Language Models for Information Retrieval
- RaFe: Ranking Feedback Improves Query Rewriting for RAG
- CoTKR: Chain-of-Thought Enhanced Knowledge Rewriting for Complex Knowledge Graph Question Answering
- Self-Consistency Improves Chain of Thought Reasoning in Language Models
- Dual Supervised Learning
- Grounding Fallacies Misrepresenting Scientific Publications in Evidence
- BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models
- SciFact-Open: Towards open-domain scientific claim verification