Is Query Likelihood Reading Word Order, or Just Counting Words?

A query likelihood model ranks documents by asking how probable the user's query would be if a language model had just read the document. This has become a strong unsupervised way to re-rank search results with open language models. The usual explanation for its strength is that a model which generates the query token by token, while reading the full document, should notice word order, negation and who-did-what-to-whom. Bag-of-words scorers like BM25 and single-vector embeddings cannot do this. Nobody has checked. The worry is concrete. A document that merely contains the query's words already makes those words cheap to predict, so much of a high query likelihood could come from vocabulary overlap rather than meaning. The benchmarks used to praise these models are also old public web text that the models may have memorized.

We propose a measurement and a correction built on one quantity. Score the query twice: once conditioned on the real document, and once conditioned on the same document with its word order destroyed. The second score is a bag-of-words counterfactual. It keeps every word the document offers and removes the structure. The gap between the two scores is the part of the relevance judgment that depends on composition. We will measure how large that part is for open base models, test whether up-weighting it fixes contrastive failures such as negation and swapped roles without damaging ordinary retrieval, and run every test on both legacy benchmarks and freshly written, post-training-cutoff minimal pairs. The fresh pairs let us separate genuine compositional reading from memorized strings.

Core Claim

Write $\ell(Q \mid D) = \frac{1}{|Q|}\sum_t \log P(q_t \mid q_{<t}, D)$ for the average per-token log-likelihood of query $Q$ given document $D$. This is the score used by UPR and by the base-LLM query likelihood models of Zhuang et al.. Let $\pi(D)$ be a within-sentence random permutation of $D$'s words. We define the compositional residual and a corrected score:

$$ r(Q,D) = \ell(Q \mid D) - \mathbb{E}_{\pi}\,\ell(Q \mid \pi(D)), \qquad s_\lambda(Q,D) = \ell(Q \mid D) + \lambda\, r(Q,D), $$

Here the expectation is estimated over $K$ permutations, and $\lambda \ge 0$ controls how much extra weight the order-dependent part receives.

We make two empirical claims. The first is that on minimal pairs, where two documents share nearly the same words but differ in meaning, raw $\ell$ from open base models will sit much closer to the bag-of-words counterfactual than the "token-level reading" story assumes. The second is that $s_\lambda$ with a small $\lambda$ chosen on held-out data will raise contrastive accuracy on contamination-controlled pairs while staying within noise on BEIR.

Either claim could fail, and both failures are informative. If $r$ is already large and $\lambda$ adds nothing, query likelihood genuinely reads structure, which is a positive result no paper in our file has established. If gains appear only on legacy NevIR and vanish on fresh pairs, the apparent compositional skill is memorization.

The correction is deliberately different from the mutual-information fix in Joint Inference of Retrieval and Generation for Passage Re-ranking. That paper divides by an unconditional prior to discount generic text. When the query is held fixed and only documents vary, a query prior $P(Q)$ is the same for every candidate and cannot change the ranking. More importantly, an unconditional prior removes generic-phrase frequency, not the credit a document earns just for containing the query's words. Our counterfactual conditions on exactly the same bag of words. It therefore removes lexical overlap specifically and leaves topical relevance from shared vocabulary in the uncorrected term.

Where This Sits

Three strands of prior work motivate the question. On the bag-of-words side, PAWS built pairs that share almost all their words but differ in meaning, and showed BERT falling from over 90% on Quora Question Pairs to 33–40%. Conceptual Contrastive Edits found that retrieval encoders barely react to word permutations or prepositions. On the Difference of BERT-style and CLIP-style Text Encoders argued that contrastive training actively rewards keyword spotting. All three examine discriminative encoders and say nothing about generative scorers.

On the lexical-overlap side, SemAntoNeg showed general-purpose embedders choosing high-overlap contradictions over true paraphrases, reaching as low as 1.9% against a 33.3% chance rate. NevIR carried the same failure into retrieval, with bi-encoders at 5–10% pairwise accuracy and MonoT5-3B at 50.6%.

On the generative side, UPR asserts that scoring $P(Q \mid D)$ avoids surface-form biases but tests only on standard QA retrieval. Zhuang et al. show base LLaMA beats instruction-tuned Alpaca on BEIR, 51.1 versus 45.1 nDCG@10. They attribute T0 and Flan-T5's earlier zero-shot results to task leakage, and they list pretraining contamination of BEIR as unresolved. Our work supplies the missing adversarial test for exactly the scorers these two papers advocate.

Two papers constrain the design. Reproducing NevIR shows that fine-tuning a retriever on NevIR lifts negation accuracy to 50.1% but collapses MS MARCO MRR@10 from 0.20 to 0.06. This is why our correction is training-free and keeps the uncorrected likelihood as its anchor. The same paper reaches 77.3% with listwise RankGPT on o3-mini. That is a strong ceiling we must report honestly. It also leaves open whether the score reflects reasoning or contamination.

Methodologically, we borrow from the contamination literature. DyVal showed that freshly generated problems expose models whose static-benchmark scores were inflated. Back to Square One showed that "group scoring," which requires both members of a twin pair to be right, deflates artifact-driven accuracy. We adopt both ideas, though DyVal's grammar-generated items are a poor fit for natural retrieval text.

Research Program

The work has three parts.

The first is the diagnostic. For each scorer and each minimal pair, we compute $\ell$ and $r$ and report what fraction of the pair-level decision margin is carried by the residual. We break this down by edit type: negation, argument-role swap, preposition change, and noun-number change. The categories follow the part-of-speech analysis of Conceptual Contrastive Edits.

The counterfactual must not simply measure how badly the model handles gibberish. The file names unnatural grammar as the central confound in order-sensitivity probes. We will therefore use three nulls of increasing gentleness: full within-sentence word shuffling, shuffling restricted to content words with function words kept in place, and sentence-order shuffling with words intact. We will report whether the conclusions agree across all three.

The second part is the correction $s_\lambda$. We select $\lambda$ on a dev split of NevIR together with an MS MARCO dev sample, using the trade-off principle from Reproducing NevIR: maximize contrastive accuracy subject to no MS MARCO loss. That $\lambda$ is then frozen for every other benchmark. With $K$ permutations, the correction costs $K$ extra forward passes. We will report accuracy as a function of $K \in \{1, 4, 16\}$ and measure wall-clock overhead. The file already flags latency as the main practical cost of generative scoring.

The third part is the contamination control. We will commission about 1,500 contrastive document pairs written from text published after the training cutoffs of every scorer. Annotators edit each source passage once, by inserting a negation, swapping two arguments, or changing a preposition. They then write two queries, each relevant to only one member of the pair. This mirrors NevIR's construction from CondaQA.

Annotators, not a model, write the queries. That keeps any generator's phrasing habits out of the likelihoods. We will also release one cohort per quarter, so that memorized templates cannot explain later results. This answers the objection the file raises against DyVal-style generators.

For legacy data, we compute each scorer's perplexity on the NevIR and BEIR documents themselves as a memorization proxy. We then test whether contrastive accuracy and the size of $r$ track that proxy. Memorized passages are a specific danger for our method: shuffling destroys a memorized string, which inflates $r$ for the wrong reason. Only the fresh pairs can rule this out.

Proposed Experiments

The scorers are the open base models evaluated by Zhuang et al. (LLaMA-7B, Falcon-7B), their instruction-tuned counterparts, a T0-style UPR scorer, and one larger and more recent open base model. All are scored pointwise with identical query-generation prompts.

The contrastive benchmarks are NevIR and our fresh pairs, both scored with pairwise accuracy at a 25% chance rate. We add a retrieval-style reformulation of PAWS-Wiki: for each source sentence that appears with both a paraphrase and a scrambled non-paraphrase, we rank the two given the source and report pairwise accuracy. ExcluIR serves as a transfer check. Standard retrieval is measured on BEIR nDCG@10 and MS MARCO dev MRR@10, re-ranking the same first-stage candidates.

Baselines

The naive reference points are chance (25% pairwise on NevIR and the fresh pairs) and BM25. BM25 is below 2% on NevIR according to NevIR, and it cannot separate PAWS word swaps by construction. These show how far a pure word-counter sits below chance on minimal pairs.

The central baseline is uncorrected pointwise query likelihood, $\lambda = 0$, from the same models. It is the method the proposal claims is partly a word-counter, and it needs no tuning. On BEIR, Zhuang et al. report 51.1 nDCG@10 for LLaMA-7B base and 55.9 for their zero-shot Falcon-7B pipeline. On NevIR we found no reported result on Lacuna for pointwise query likelihood scoring, so we will produce it ourselves.

The second baseline is query likelihood interpolated with BM25, as in Zhuang et al., tuned on the same dev data as $\lambda$. It is the obvious hybrid a practitioner would deploy.

The third is mutual-information correction in the style of JPR, which reports 51.2 nDCG@10 on BEIR against 49.4 for its BERT-FT baseline. We implement its prior term with the same scorer and the same tuning budget.

The strongest supervised pointwise competitor is MonoT5-3B, at 50.6% NevIR pairwise accuracy per NevIR and 56.7 BEIR nDCG@10 per Zhuang et al.. The strongest system of any kind is listwise RankGPT with o3-mini, at 77.3% NevIR pairwise accuracy per Reproducing NevIR. We will rerun it on the fresh pairs.

The key ablation replaces the order-destroyed counterfactual $\pi(D)$ with a length-matched random document. This tests whether any subtracted baseline helps, or only one that holds the bag of words fixed. We also report the residual $r$ used alone.

Beating the comparisons means three things. On fresh pairs, $s_\lambda$ must exceed both uncorrected query likelihood and MonoT5-3B under group scoring, with a paired bootstrap interval excluding zero. It must stay within one nDCG@10 point of uncorrected query likelihood on BEIR. And it must beat the random-document ablation. We do not expect to beat o3-mini RankGPT outright. The relevant comparison there is how much each system's accuracy drops from NevIR to fresh pairs.

Why not just…?

Why not just use listwise RankGPT, which already reaches 77.3% on NevIR? It is the strongest number in the file, and we treat it as a real threat rather than a strawman. Its limits are practical and epistemic. It requires a frontier API, offers no token-level likelihoods (the file's logit-access constraint cuts the other way here), and its NevIR score comes from public Wikipedia-derived text it may have seen. If RankGPT holds 77% on our fresh pairs while corrected query likelihood with open models stays far below, we will report that open pointwise scoring cannot close the gap. The diagnostic findings still stand.

Why not just interpolate with BM25? Interpolation adds more word-counting. On NevIR, BM25-style sparse scoring is below 2% because it rewards the document that contains the query's words, which on negation pairs is often the wrong one (NevIR). On PAWS-style swaps it is exactly tied. We include it as a baseline, and we expect it to lower contrastive accuracy relative to raw query likelihood.

Why not just apply the mutual-information correction from JPR? For a fixed query, dividing by the query's prior probability leaves the document ranking unchanged. JPR's passage-prior variant discounts generic documents, not documents that share the query's vocabulary. Our ablation runs both under the same budget. If JPR matches $s_\lambda$ on fresh pairs, our argument that the counterfactual must hold the bag of words fixed is wrong, and we will say so.

Why not just fine-tune on NevIR or PAWS? Reproducing NevIR shows this trades MS MARCO MRR@10 from 0.20 down to 0.06 and transfers nothing to ExcluIR. Our correction is training-free and anchored on the uncorrected score. ExcluIR results will show whether it transfers where fine-tuning did not.

Why not just use an NLI-tuned encoder? NLI tuning reaches 93.6% on SemAntoNeg (SemAntoNeg), but that is sentence selection, not retrieval. Single-vector pooling discards order, as C-PHRASE discusses. We will run one NLI-tuned bi-encoder on NevIR and the fresh pairs to make the comparison direct.

Decisive Checks

Three results would settle the proposal.

First, on fresh pairs and under group scoring, does uncorrected query likelihood fall well below MonoT5-3B, with $r$ carrying only a small share of its decision margin? If so, the token-level-reading story is wrong for open base models.

Second, does $s_\lambda$ raise fresh-pair accuracy while holding BEIR within one nDCG@10 point, and does it beat the random-document ablation? If so, the bag-of-words counterfactual is the working ingredient.

Third, is any system's NevIR-to-fresh drop correlated with its perplexity on NevIR documents? If so, part of what the field has read as negation understanding in retrieval is memorization, and the dated fresh pairs become the benchmark this area should report.