Neural search engines, the systems that encode queries and documents as vectors and rank by similarity, tend to rank text written by large language models (LLMs) above human text that says the same thing. This preference is called source bias. Separately, every search system struggles with the vocabulary mismatch problem, or lexical gap: people describe the same thing in different words than document authors do. This proposal argues that the two problems are linked. Source bias, we hypothesize, is mostly a fallback behavior. When a query and a relevant human document share few words, the retriever has little real matching evidence, so it leans on a cheap prior that favors fluent, predictable, LLM-like prose. Two consequences follow, and both can be tested. First, source bias should grow as the lexical gap widens. Second, the standard fix for the gap, having an LLM expand or rewrite the query, may close the gap by moving the query toward the machine's register, so that the query lands closer to synthetic documents than to human ones.
This matters because both trends are accelerating. LLM text is filling search indices, and LLM-based query rewriting is becoming the default way to handle lexical mismatch in retrieval-augmented generation. If the gap-closing tool makes the bias worse, the web's human authors lose twice, and they lose most in specialized domains such as medicine, science, and law, where the gap is widest. If the hypothesis holds, it also points to a specific correction: debiasing should be strongest where lexical evidence is weakest, not applied uniformly.
Core Claim
The claim is that the size of neural source bias depends on how much lexical evidence connects the query to the human document. Bias should be small when the overlap is high and large when it is near zero. Because of this, a correction whose strength depends on query–document lexical overlap should remove more bias than a single global correction, and lose less accuracy on technical, jargon-heavy collections. A corollary is that LLM query expansion fed to a dense retriever will raise recall and also push results toward synthetic documents. The same expansion fed only to a lexical channel will not.
The motivating evidence is spread across papers that never measured this interaction. Neural Retrievers are Biased Towards LLM-Generated Content rewrites relevant human documents with an LLM and finds opposite behavior from the two model families on SciFact: ANCE has a Relative Δ of −47.0%, meaning it prefers the synthetic copy, while BM25 has +23.9%, meaning it prefers the human one. The authors attribute BM25's preference to human authors' specific word choices. So the neural preference for LLM text appears even though the human text carries more exact query-term evidence. That is what a fallback prior would produce. Perplexity Trap shows that dense retrievers treat low perplexity, a measure of how predictable text is to a language model, as a shortcut for relevance, with correlations below −0.8 on DL19, TREC-COVID and SCIDOCS. Data, Not Model traces the shortcut to MS MARCO training pairs, where positives have lower perplexity than mined negatives. That paper also reports that LLM text has slightly higher inverse document frequency (IDF), a measure of how rare and specific its words are. This might look like it contradicts our story. We read the two findings as compatible: corpus-level term rarity is not the same thing as overlap with a particular query's terms, and BM25's human preference shows that query-specific overlap favors human text. Our stratified audit separates the two directly. Finally, BEIR shows that MS MARCO–trained dense retrievers often fall below BM25 on mismatch-heavy out-of-domain sets. That fits the idea that under a wide gap they rely on priors rather than matching.
Research Program
The first component is a gap-stratified audit of source bias. We use the paired human/LLM corpora of Cocktail, which covers 16 datasets including NQ-UTD, a post-cutoff set that rules out memorization. We also use SciFact+AIGC and NQ320K+AIGC from Neural Retrievers are Biased Towards LLM-Generated Content. For each relevant query–document pair, we measure the lexical gap as the fraction of the query's content terms that are absent from the human original, with IDF-weighted and lemmatized variants. We then compute Relative Δ and NDCG@k separately within each overlap bucket. We do this for BM25, for the dense retrievers ANCE, TAS-B and Contriever, for ColBERT (a late-interaction model that matches individual query and document tokens), and for a monoT5 cross-encoder re-ranker. Observational buckets are confounded by domain and query type. We therefore add an intervention on the query side that holds the information need fixed and creates a controlled gap ladder. At each step we mechanically drop document-shared terms from the query or replace them with dictionary synonyms. We deliberately do not ask an LLM to paraphrase the query, since that would inject machine register and confound the very effect we want to isolate. The prediction is that the neural Relative Δ becomes more negative as overlap falls, while BM25 mostly loses recall without changing which source it prefers.
The second component is a controlled expansion experiment. We apply Query2Doc and chain-of-thought expansion from Query Expansion by Prompting Large Language Models, plus a RaFe-style rewriter from RaFe trained with reward from a BGE re-ranker. The original expansion study only fed BM25, and RaFe never inspected which source its retrieved documents came from. We route each expansion three ways: into BM25 only, into the dense encoder only, and into both. We measure recall and source bias per gap bucket for each route. For the RaFe-style rewriter, we additionally compare a rewriter trained with reward computed only over human documents against one trained over the mixed corpus. This tests whether the rewriter learns to target the re-ranker's synthetic preference.
The third component is the correction itself. CDC, the inference-time causal correction from Perplexity Trap, subtracts an estimated perplexity effect from the relevance score with a single coefficient $\beta$. We let that coefficient depend on lexical evidence:
$$s'(q,d) = s(q,d) - \beta(g_{qd})\,\mathrm{PPL}(d),$$
where $s(q,d)$ is the retriever's score for query $q$ and document $d$, $\mathrm{PPL}(d)$ is the document's perplexity under a reference language model, and $g_{qd}$ is the query–document lexical gap. We parameterize $\beta(g)$ as a monotone piecewise-linear function. It is estimated with the same instrumental-variable regression as CDC, using source type as the instrument, but fitted within gap buckets on a held-out calibration split. If the core claim is false, the fitted $\beta(g)$ will come out flat and the method reduces to CDC. That makes the correction self-falsifying.
Baselines
The naive reference point is the unmodified dense retriever or re-ranker run on the mixed corpus. It anchors the scale: Cocktail reports an average Relative Δ of roughly −25% across neural models, and ANCE reaches −47.0% on SciFact (Neural Retrievers are Biased Towards LLM-Generated Content). BM25 is the lexical reference, with +23.9% on SciFact, which shows that preferring human text is achievable but at a cost in semantic recall.
The strongest existing debiasing competitors are global CDC and the two fixes from Data, Not Model. The first fix retrains with only in-batch negatives, which moved average ANDSR@5 on the Cocktail subset from −0.099 to −0.024 with small effectiveness loss. The second is an inference-time projection that removes the embedding direction separating LLM from human text. The file gives no reported magnitude for CDC's bias reduction; no reported result found on Lacuna for CDC's debiasing effect on Cocktail. We will therefore reproduce CDC ourselves. All corrections are applied to the same base checkpoints, use the same perplexity scorer, and are calibrated on the same held-out queries. The in-batch-negative model is trained with the same compute as our base model. Our method counts as beating them if two things hold. First, it reaches equal or better average Relative Δ, matching or improving on the −0.024 ANDSR@5 level. Second, it shows statistically smaller NDCG@10 loss on the high-gap buckets of TREC-COVID, SciFact and SCIDOCS.
For the expansion arm, the strongest gap-closing baseline is chain-of-thought expansion into BM25, which reached Recall@1K of 90.61 on MS MARCO against 87.82 for BM25 and 88.68 for pseudo-relevance feedback. We report recall alongside source bias for every route. A configuration counts as better only if it keeps most of that recall gain without shifting the human/synthetic ratio in the top 10.
The key ablation replaces the gap-dependent $\beta(g)$ with a constant fitted on the same data, and separately with a $\beta$ that depends on query length instead of overlap. This isolates lexical evidence as the active ingredient.
Why not just…?
Why not just use BM25 or a fixed BM25–dense hybrid? BM25 avoids neural source bias, but BEIR and Yang et al. ( WikiQA: A Challenge Dataset for Open-Domain Question Answering) show it fails exactly when queries share no words with relevant documents. A fixed-weight hybrid is a real threat. It is a crude form of evidence-dependent weighting, and it might capture much of our effect. We include linear interpolation tuned on the same calibration split. If it matches gap-conditioned CDC on both bias and high-gap NDCG, we will report that the simple hybrid suffices.
Why not just use pseudo-relevance feedback instead of LLM expansion? Feedback terms come from human documents, so they should not inject machine register. However, under a wide gap the first-round results are noise, and in the original expansion study feedback trails chain-of-thought expansion (88.68 vs 90.61 Recall@1K on MS MARCO). We include RM3-style feedback in the expansion arm. The per-bucket results will show whether it avoids bias amplification only where it also fails to retrieve.
Why not just filter synthetic documents with an AI-text detector? Detection of LLM text at the level of single documents is increasingly unreliable (LLM-Generated Fake News Induces Truth Decay). Many synthetic documents are also accurate and useful. The goal is to rank by relevance regardless of source, not to exclude machine text.
Why not just apply global CDC everywhere? This is the most direct competitor. If source bias does not vary with the lexical gap, global CDC is the right answer and our fitted $\beta(g)$ will be flat. The audit and the constant-$\beta$ ablation settle this directly. We also expect a second difference. A global coefficient fitted mostly on low-gap queries should under-correct high-gap queries. A coefficient fitted on high-gap queries should over-penalize technical text, whose perplexity is naturally high, which is the concern raised as Gap 5 in the file. A gap-dependent coefficient avoids both.
Why not just keep LLM expansion out of the dense channel? That routing is one of our experimental conditions, not a competitor we argue against. If it removes the amplification while keeping recall, it becomes a cheap practical recommendation and one of the findings of the study.
Decisive Checks
The idea stands or falls on three results. The first is whether, on the mechanical gap ladder, neural Relative Δ grows more negative as query–document overlap falls, while BM25's source preference stays roughly constant. A flat curve refutes the fallback hypothesis. The second is whether chain-of-thought or Query2Doc expansion routed into dense retrievers shifts the top-10 results toward synthetic documents more than the same expansion routed into BM25, on NQ-UTD where memorization is excluded. The third is whether the fitted $\beta(g)$ is clearly non-constant and beats both constant-$\beta$ CDC and the tuned hybrid on high-gap TREC-COVID and SciFact queries, measured on held-out NDCG@10 together with Relative Δ. A positive answer on all three would show that source bias and vocabulary mismatch are one problem seen from two sides. A negative answer on the first would still be useful, because it would establish that existing global corrections are adequate across lexical regimes.
References
- Neural Retrievers are Biased Towards LLM-Generated Content
- Perplexity Trap: PLM-Based Retrievers Overrate Low Perplexity Documents
- Data, Not Model: Explaining Bias toward LLM Texts in Neural Retrievers
- BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models
- Cocktail: A Comprehensive Information Retrieval Benchmark with LLM-Generated Documents Integration
- Query Expansion by Prompting Large Language Models
- RaFe: Ranking Feedback Improves Query Rewriting for RAG
- WikiQA: A Challenge Dataset for Open-Domain Question Answering
- WikiQA: A Challenge Dataset for Open-Domain Question Answering
- LLM-Generated Fake News Induces Truth Decay in News Ecosystem: A Case Study on Neural News Recommendation