Retrieval-augmented generation (RAG) means handing a language model retrieved documents alongside a question. When the right answer sits among irrelevant or misleading passages, the model is often distracted by them, and it is especially likely to miss evidence placed in the middle of a long prompt. The strongest fixes either train a separate 7-billion-parameter model to compress the documents, or require reading the reader model's internal token probabilities. Commercial APIs do not expose those probabilities. This note asks a narrower and cheaper question: how much of what makes a retrieved sentence help or hurt a frozen reader can be predicted from plain corpus statistics, before the reader sees anything?
The proposal splits each sentence's relevance to the question into two parts. The first is surface overlap: how many of the question's rare words the sentence repeats, measured with TF-IDF, a classic weighting that scores a word by how often it appears locally and how rare it is across the corpus. The second is associative support: how strongly the sentence's other words co-occur with the question's words across the corpus, measured with pointwise mutual information (PMI), the log ratio of how often two words appear together to how often they would by chance. The hypothesis is that distracting sentences have high overlap but little associative support, because they echo the question without adding anything the question points to. Answer-bearing sentences contain new words that the corpus strongly ties to the question, for example "Shakespeare" alongside "Hamlet". If this holds, a filter that runs on a single CPU and needs no access to the model's internals can do much of the work of neural relevance scoring.
Core Claim
The claim has two parts, and both can be tested. The first is diagnostic. Take sentence-level neural PMI in the style of Pointwise Mutual Information as a Performance Gauge for Retrieval-Augmented Generation, meaning how much adding a sentence raises $\log p(q|c)/p(q)$ for the question $q$ and context $c$. The claim is that associative support explains a substantial share of that quantity, and of whether a sentence flips a correct answer to a wrong one, beyond what surface overlap and dense embedding similarity already explain. Surface overlap on its own should be positively associated with harm once associative support is held fixed. The second part is practical. Use the two scores to prune sentences and then place the survivors in the positions the reader attends to best. This should recover a large fraction of the white-box neural-PMI gains on black-box readers at negligible cost.
Concretely, for question terms $Q$ and sentence terms $S$, surface overlap is $O(s)=\sum_{t\in S\cap Q}\mathrm{idf}(t)$, and associative support is
$$A(s)=\frac{1}{\sqrt{|S|}}\sum_{u\in S\setminus Q}\mathrm{idf}(u)\,\max_{t\in Q}\,\max\big(0,\mathrm{PMI}(t,u)\big),$$
Here PMI is computed from windowed co-occurrence counts over the retrieval corpus. The count threshold (pairs seen at least five times) and the window follow Word Association Norms, Mutual Information and Lexicography, which introduced windowed PMI for collocations and documented its instability on rare pairs. The square-root length term keeps long sentences from winning by volume, which addresses the length sensitivity known for raw TF weighting. The score deliberately sums only over words not in the question. That exclusion is the whole difference from lexical retrieval, and a later ablation removes it.
Relation to Prior Work
Two papers motivate the gap. Pointwise Mutual Information as a Performance Gauge for Retrieval-Augmented Generation shows that query–context neural PMI tracks accuracy along the same U-shaped curve across prompt positions. It uses that finding to reorder whole documents, placing high-PMI ones at the start and end of the prompt ("search by curvature"). That method needs token log-probabilities and works only at the document level. We reuse its placement rule as-is, but we replace the neural score with corpus statistics and work at the sentence level. Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering shows that readers collapse as retrieval deepens to 100 documents. Its fix penalizes answers using $P(\text{query}|\text{document})$, which omits the marginal $P(\text{query})$. Our associative score normalizes by marginal frequency by construction, since that normalization is what makes it PMI.
Three trained systems set the bar the proposal must be judged against, and each has a different weakness. An Information Bottleneck Perspective for Effective Noise Filtering on Retrieval-Augmented Generation trains a 7B filter model with Monte Carlo sampling and preference optimization. CompAct: Compressing Retrieved Documents Actively for Question Answering trains a Mistral-7B model on GPT-4o trajectories to summarize segments one at a time. Aligning Extraction and Generation for Robust Retrieval-Augmented Generation fine-tunes the reader itself to extract and then answer. It also argues that filters applied before generation are fundamentally limited, and that argument constrains us. If Ext2Gen is right, a statistical pre-filter should plateau well below it, and we treat that outcome as possible rather than dismissing it.
The only prior work that feeds corpus PMI into transformers is Enhancing Self-Attention with Knowledge-Assisted Attention Maps. It fuses PMI term-correlation maps into BERT attention to help with rare words. That approach requires architectural changes and retraining, and it was tested on short inputs. We use PMI in the opposite role, outside the model and for removing text rather than enriching it. Validation and Evaluation of Automatically Acquired Multiword Expressions for Grammar Engineering found that mutual information separates real associations where chi-square fails. That result supports using PMI rather than a simpler co-occurrence test.
Research Program
The first stage is the diagnostic study, and it decides whether the filtering stage is worth building. We draw questions from NQ-Open, TriviaQA and HotpotQA, and retrieve the top 20 documents for each. For every sentence, we measure three things with an open reader (Mistral-7B, matching Liu et al.). The first is the sentence's marginal neural-PMI gain. The second is its flip effect: whether inserting it next to the gold sentence changes a correct answer to an incorrect one. The third is whether it contains the gold answer. We regress each of these on $O$, $A$, dense cosine similarity, sentence length and position. The central quantities are the partial correlation of $A$ after controlling for $O$ and dense similarity, and the sign of $O$ when predicting flips.
The second stage is the filter. We keep sentences with high $A$, subject to a minimum $O$ so that obviously off-topic text is dropped. We then group the survivors by source document and place them using the curvature rule from Pointwise Mutual Information as a Performance Gauge for Retrieval-Augmented Generation, with $A$ standing in for neural PMI. We tune the thresholds for each dataset on a development split of 500 questions, and all other baselines get the same budget. The filter is then run in front of an open reader and a black-box API reader, where no internal probabilities are available.
Baselines
The naive reference points are "no retrieval" and "all retrieved documents concatenated in retriever order". These matter because An Information Bottleneck Perspective for Effective Noise Filtering on Retrieval-Augmented Generation reports that top-5 retrieval without filtering scores 14.7 exact match (EM) on NQ, below the 16.2 EM of no retrieval. Any filter has to clear the unfiltered prompt on the same retrieved set.
The strongest training-free competitor is Liu et al.'s neural-PMI search by curvature. They report NQ-Open accuracy rising from 62.89% to 65.72% on Mistral-7B. We rerun it on the same retrieved documents and the same reader. We also give it a sentence-level variant, so that granularity is not what separates it from our method. Because it requires white-box access, it serves as the upper reference for the open reader and cannot be run on the API reader. Beating it is not required. Recovering at least half of its gain over the unfiltered prompt at a small fraction of its cost would count as success, and so would matching it within noise on the black-box reader, using its open-model scores as the comparison.
Among trained systems, the information-bottleneck filter is the strongest compressor in the file on NQ, at 21.5 EM with the same 7B model setup. CompAct reports +7.0 F1 over RECOMP on HotpotQA. Ext2Gen reports 0.557 against 0.473 for Llama3.1-8B-Instruct on its noisy test split. We run released checkpoints where they are available and use the papers' own evaluation protocols. We compare against their reported numbers only on matching splits, and we say so explicitly wherever the setups differ. We do not expect to beat these systems outright. The question is where our method sits on the accuracy-versus-cost curve.
Two ablations isolate the key ingredient. The first is an overlap-only filter, which ranks sentences by BM25 or $O$. The second is a support-without-exclusion variant, which sums PMI over all sentence words, including those that already appear in the question. If the gains disappear when question words are allowed back into the associative score, then the claimed "new associated content" signal is really just lexical matching.
Why not just…?
Why not just rank sentences with BM25? This is the most direct threat, and the ablation above tests it. Our hypothesis predicts that BM25 fails in a specific way. It will rank highly the distractors that repeat the question's entities, because repeated entities are exactly what makes those sentences distracting. The same failure shows up at the document level in the reported collapse of lexically and densely retrieved contexts at depth 100 in Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering. We will report results separately on a "hard distractor" subset: questions where BM25's top sentence does not contain the answer. If BM25 matches our filter on that subset, the associative component adds nothing, and we will report the method as unnecessary.
Why not just compute neural PMI with a small open proxy model? A 1B-parameter open model could compute Liu-style sentence PMI for a black-box reader, and this could plausibly work as well as our method. We therefore treat it as a full competitor, not a straw man, and we match it in the diagnostic stage by measuring how well each score agrees with the large reader's own PMI. If the proxy model wins clearly, the practical conclusion changes. The lexical score would then be useful mainly as a cheap first pass that reduces how many sentences the proxy must score, and we will measure that combination as well. The diagnostic finding about whether distraction is lexical still holds either way.
Why not just tell the model to answer "unanswerable" when unsure? Cho et al. show that this instruction alone falls short without the score adjustment. It also does nothing about evidence buried mid-prompt, which is the positional effect that Liu et al. document. We include the instruction in every condition, including ours, so that it cannot account for any difference between methods.
Why not just retrieve fewer documents? Using a small top-k trades distraction for recall. CompAct and R²AG both document degradation as depth grows, and a shallower retrieval simply loses the answer on multi-hop HotpotQA questions. We sweep k over 3, 5, 10 and 20 for the unfiltered baseline and report the best k per dataset, so our filter is compared against the best achievable truncation.
Decisive Checks
The diagnostic stage succeeds if associative support has a clearly nonzero partial correlation with neural-PMI gain after controlling for overlap and dense similarity on all three datasets, and if overlap, holding support fixed, predicts answer flips positively. If $A$ adds nothing beyond $O$, we stop and report a negative result: distraction in these readers is not captured by corpus association.
The filtering stage has three evaluation settings. The first is SQuAD with 100 retrieved documents, the high-noise setting from Cho et al. The second is the Ext2Gen noisy split with up to 25 distractor chunks. The third is NQ-Open with Mistral-7B, which gives a direct comparison to 65.72%. In each, we report exact match or accuracy, input tokens per question, and preprocessing time on a single CPU. A final check tests whether $A$ reproduces the U-shaped curve across positions that Liu et al. found for neural PMI when a gold sentence is moved through the prompt. If it does, the placement rule transfers to black-box readers for a principled reason and not by coincidence.
References
- Pointwise Mutual Information as a Performance Gauge for Retrieval-Augmented Generation
- Word Association Norms, Mutual Information and Lexicography
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
- An Information Bottleneck Perspective for Effective Noise Filtering on Retrieval-Augmented Generation
- CompAct: Compressing Retrieved Documents Actively for Question Answering
- Aligning Extraction and Generation for Robust Retrieval-Augmented Generation
- Enhancing Self-Attention with Knowledge-Assisted Attention Maps
- Validation and Evaluation of Automatically Acquired Multiword Expressions for Grammar Engineering