When a question-answering system built on a language model retrieves passages, some scoring function decides which passages enter the model's context. Two older literatures tell us that function is biased in predictable ways. Similarity scores between a short text and a long one are distorted by length alone. Dense vector similarity shrinks as a passage grows, because extra content "dilutes" its embedding. Word-overlap scores do the opposite and grow, because longer passages share more words by chance. Separately, correct evidence often shares no content words with the question. The third literature, on distraction, shows that readers fail badly when the wrong passages get in. Reasoning models are hit hardest: they spend their extended "thinking" justifying a misleading snippet instead of rejecting it. No one has asked whether the first two biases are the cause of the third. This proposal argues they are a large part of it. Admission scores systematically let in long, word-overlapping distractors and keep out short or differently-worded gold evidence, so readers are distracted even when the retriever "found" the answer somewhere.
The plan has two steps. First, build a controlled perturbation benchmark that changes a passage's length and its word overlap with the question while keeping its meaning fixed. With it we measure how much each common scorer moves, and whether those movements predict reader errors and runaway reasoning. Second, test a deliberately cheap fix: admit passages by how their score compares with same-length background passages scored against the same question, instead of by raw score. If length calibration accounts for a meaningful share of the distraction that heavier methods address, the field has been training large extractors and compressors to undo a bias introduced one step earlier.
Core Claim
The core claim is that reader distraction in retrieval-augmented question answering is partly an artifact of length-confounded admission scoring, not only of noise volume. The bias runs in two directions. Under dense scoring, long gold passages are under-admitted (dilution). Under sparse and query-likelihood scoring, long distractors are over-admitted (inflation). Vocabulary mismatch compounds both, because gold passages written in different words lose whatever overlap signal they had. We predict that the misleading-snippet failure is strongest when the misleading text is long and word-overlapping, and that a length-calibrated admission rule removes a measurable fraction of it at negligible cost.
Grounding in Prior Work
The motivating evidence comes from the similarity literature. Text Reuse Detection using a Composition of Text Similarity Measures showed that padding text with peripheral detail dilutes the share of core content. Unsupervised Concept Representation Learning for Length-Varying Text Similarity showed that long documents' global vectors drift from concise equivalents. It reports raw BERT reaching an MRR (mean reciprocal rank, a ranking metric) of only 0.181 on CL-SciSumm 2017 against 0.944 for its concept-matching method. Listening Comprehension over Argumentative Content observed the opposite failure: overlap-based matchers prefer longer candidate sentences. A Sentence is Worth 128 Pseudo Tokens showed that contrastive sentence encoders such as SimCSE learn length as a similarity shortcut. Its fixed-size bottleneck lifts a length-adversarial STS subset from 71.08% to 74.44% Spearman. These works fix the similarity function for matching tasks. None of them asks what the residual bias does once the scores gate a reader's context. That downstream question is ours. DeepPurple adds that human similarity judgments shift from lexical to topical as texts lengthen. That gives a principled reason to expect no single uncalibrated score to behave consistently across lengths.
The distraction literature supplies the failure we try to explain. Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents (DAS) shows zero-shot readers collapsing as retrieval depth reaches 100 documents, with wrong answers given confidence up to 0.98. It repairs this with an abstention instruction and a rescoring by $P(\text{query}\mid\text{document})$. Thinking in a Crowd documents the "rationalization trap": Qwen3-8B falls from 96.22% to 77.03% with two misleading snippets in thinking mode, below its 74.87% without thinking. Both treat the passages as given. Our contribution is to vary the passages' length and wording while holding their meaning fixed, which neither study does.
One finding constrains us directly. DAS's query-likelihood score is itself a lexical signal, so we predict it inherits both inflation and the vocabulary-gap penalty. Yang et al. report that many correct answers share zero content words with the question. If our prediction holds, part of DAS's large gain comes from removing obviously irrelevant documents, while terse or differently-worded gold evidence stays at risk. If it fails, that is informative too.
The closest existing work on mechanisms sits downstream of admission. CompAct compresses retrieved sets iteratively. Aligning Extraction and Generation for Robust Retrieval-Augmented Generation (Ext2Gen) trains the reader by preference optimization to extract key sentences before answering. Mix-of-Granularity learns which chunk size to retrieve per query. These are the competitors we must beat or match. We differ in three ways: we diagnose the scorer's length bias directly, we intervene before the reader, and our intervention needs no training of the reader.
Research Program
The first component is the perturbation benchmark. We start from items whose evidence roles are already labeled. SciAux, derived from ScienceQA and cataloged as billhdzhao/SciAux, has 6,828 questions with helpful, irrelevant and misleading snippets. HotpotQA and Natural Questions have gold passages and supply retrieved distractors.
For every snippet we generate content-preserving variants along two axes. Length takes three levels: condensed to a single sentence, original, and expanded with on-topic but answer-neutral elaboration. Lexical overlap takes two levels: original wording, and a paraphrase that removes every content word shared with the question. A language-model paraphraser produces the variants. They are kept only if a separate entailment check in both directions and a 300-item human audit confirm that meaning is unchanged. Expansion text is also checked so that it introduces no claim bearing on the answer.
Crossing relevance role, length and overlap gives a factorial design. In it, any change in a scorer's ranking or a reader's answer can be attributed to form rather than content.
The second component is the measurement. For each scorer we record how far its score moves across variants of the same content. The scorers are BM25, a dense bi-encoder, PT-BERT-style bottleneck embeddings, a cross-encoder reranker, and DAS's query likelihood. We then fix the number of admitted passages and let each scorer choose them. For each reader, we record accuracy, abstention rate, and thinking-token length. The readers are Qwen3-8B with thinking on and off, and Llama3.1-8b-instruct.
The key analysis is a mediation test. Does the scorer-induced change in which passages are admitted explain the change in reader accuracy? We compare it against a control where the same passages are forced in regardless of score. That separates "the scorer let the wrong thing in" from "the reader is distracted by long text no matter how it got there." The second effect is the one Thinking in a Crowd already shows. The first is new.
The third component is the fix. For a query $q$ and passage $d$ with length bin $\ell(d)$, we replace the raw score $s(q,d)$ by
$$z(q,d) = \frac{s(q,d) - \mu_q(\ell(d))}{\sigma_q(\ell(d))},$$
where $\mu_q(\ell)$ and $\sigma_q(\ell)$ are the mean and standard deviation of $s(q,d')$ over a fixed sample of random corpus passages $d'$ in the same length bin. The z-score therefore asks how unusual this passage's score is for a passage of its length against this query. That directly cancels the dilution penalty and the inflation reward.
Passages are admitted if $z$ exceeds a threshold tuned on a development split. If none qualifies, the reader receives the abstention instruction alone, which ties this step to DAS's abstention mechanism. The cost is scoring roughly 200 background passages per query. A cheaper variant fits a query-independent length correction offline, and serves as an ablation.
To handle vocabulary mismatch, we compute $z$ separately for a sparse and a dense scorer and admit on the maximum of the two. The paraphrased zero-overlap gold passages test whether the dense channel rescues what the sparse channel loses.
Baselines
The naive reference point is uncalibrated top-k admission from each scorer feeding a zero-shot reader. This is the setting in which DAS reports collapse and in which SciAux's misleading condition was measured. Every other system is compared against it, at the same k and the same readers.
The strongest training-free competitor is DAS from Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents. The file reports a 64% average increase in Exact Match over naive zero-shot readers and a 231% gain on SQuAD with DPR and 100 documents. Like our method, it acts at the admission and answer-selection stage without retraining. We run it with the same retrievers, the same 100-document depth on NQ and SQuAD, and the same readers. Beating it means higher Exact Match at equal depth, and specifically on the zero-overlap and short-gold cells, where we predict its query-likelihood score fails.
The strongest trained competitor is Ext2Gen from Aligning Extraction and Generation for Robust Retrieval-Augmented Generation. On Llama3.1-8b-instruct it reports an average score of 0.557 against 0.473 for the default model. It reportedly outperforms independent compressors including CompAct, which itself reports a 7.0 F1 gain over RECOMP on HotpotQA. We use the released Ext2Gen setting on the same base model and the same retrieved lists, with our admission rule placed in front of the default reader.
We do not expect a training-free gate to beat a preference-trained reader overall. Success has two parts. First, calibrated admission alone should close at least half the gap between the default model and Ext2Gen. Second, calibration combined with Ext2Gen should improve on Ext2Gen alone, which would show the two fix different errors. For reasoning models, the reference is SciAux's 77.03% for Qwen3-8B with thinking and two misleading snippets. Here success means recovering accuracy above the 74.87% thinking-off level while also shortening rationalization traces.
Three ablations isolate the key ingredient. Raw scores with the same threshold-and-abstain logic test whether calibration matters or only abstention does. The query-independent length correction tests whether per-query statistics are needed. Fixed-length windowing of all passages before scoring removes length variation by construction, and is discussed below.
Why not just…?
Why not just chunk everything to a fixed length? If every candidate passage has the same length, the length confound among candidates disappears, and this is a real threat. However, Mix-of-Granularity reports that the best chunk size varies by query and that fixed chunking underperforms dynamic selection. Unsupervised Concept Representation Learning for Length-Varying Text Similarity argues that short units are exactly where vocabulary mismatch bites hardest, because they lack surrounding context. Fixed windowing is therefore one of our ablations, evaluated in the zero-overlap cells. If it matches calibrated admission there, we will report that the diagnosis holds and the remedy is simply fixed-length chunking. That is a useful, cheaper conclusion, and we would not oversell the z-score.
Why not just turn thinking off, or add the abstention prompt? Thinking in a Crowd shows that thinking-off barely avoids the trap: 74.87% versus 77.03%, both far below 96.22%. Disabling reasoning also forfeits the 10 to 13% gain it gives on clean context. The abstention instruction is part of DAS, which is already a baseline. Our factorial design reports both switches in every cell, so we can see whether they help uniformly or only where admission already erred.
Why not just let a trained extractor like Ext2Gen or CompAct filter the noise? Extraction can only keep what was admitted. If short, differently-worded gold was never admitted, there is nothing to extract, and the file notes that both methods inherit their retriever's lexical failures. Their reported gains could still absorb most of what we measure. The combined condition, calibration plus Ext2Gen, settles this. No additive gain would mean downstream extraction already compensates.
Why not just use a frontier model as a reranker? SMILE reports that GPT-4o is itself prone to favoring verbose, fluent text, and that it is roughly nine times slower than a lightweight metric. We include a model-based reranker among the measured scorers to test whether it shows the same length sensitivity, rather than assuming it does.
Decisive Checks
The idea stands or falls on three results. First, content-preserving length and overlap perturbations must change admission decisions for at least some widely used scorers, by an amount larger than the variation between paraphrases of equal length. Without that, there is no confound to fix. Second, in the forced-admission control, the scorer-induced share of reader errors must be non-trivial. If readers fail equally whichever way the passages got in, the problem lives in the reader, and the existing reader-side methods are the right investment. Third, calibrated admission must improve over raw admission and over DAS on the zero-overlap and long-distractor cells, at equal retrieval depth and equal reader. It must also reduce the thinking-token inflation that marks rationalization on SciAux. Evaluation uses Exact Match and F1 on NQ, SQuAD and HotpotQA, accuracy and mean reasoning length on SciAux, and admission recall of gold passages per factorial cell. A negative result on the third check paired with a positive result on the first two would still be publishable: it would show that the bias is real but is better absorbed downstream.
References
- Text Reuse Detection using a Composition of Text Similarity Measures
- Unsupervised Concept Representation Learning for Length-Varying Text Similarity
- Listening Comprehension over Argumentative Content
- A Sentence is Worth 128 Pseudo Tokens: A Semantic-Aware Contrastive Learning Framework for Sentence Embeddings
- DeepPurple: Estimating Sentence Semantic Similarity using N-gram Regression Models and Web Snippets
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
- Thinking in a Crowd: How Auxiliary Information Shapes LLM Reasoning
- WikiQA: A Challenge Dataset for Open-Domain Question Answering
- CompAct: Compressing Retrieved Documents Actively for Question Answering
- Aligning Extraction and Generation for Robust Retrieval-Augmented Generation
- Mix-of-Granularity: Optimize the Chunking Granularity for Retrieval-Augmented Generation
- SMILE: A Composite Lexical-Semantic Metric for Question-Answering Evaluation