When a language model answers a question from retrieved documents, the passages that mislead it are rarely random. They usually mention the same people, places, or dates as the question, but in a sentence that does not actually relate them: a birth year in an aside, a company name in a footer, a drug and a gene that share a methods paragraph. Classical text mining has a name for this, co-occurrence noise: treating "these words appear near each other" as if it meant "these things are related." It also has a remedy. Check whether the words are grammatically or rhetorically bound together before trusting the association. The newer literature on contextual distraction, meaning the tendency of large language models (LLMs) to be pulled off course by extraneous text in their prompt, mostly tests models against irrelevant or loosely topical passages. It has not isolated this specific kind of lure.
This note proposes to measure and then remove that failure. The claim is that much of the harm retrieved context does to a reader comes from co-occurrence lures: sentences that contain the question's entities together with an answer-typed candidate, but without a relation that binds them. We further claim that a filter which keeps evidence only when the question's entities are bound to the candidate removes much of this harm. A binding filter accepts a sentence only if the entities are linked by a dependency path, or sit inside one connected chain of discourse units. Ordinary relevance and similarity scores cannot do this, because they reward exactly the overlap that makes a lure convincing. The method runs before the reader, so it works with black-box APIs.
Core Claim
The claim has two testable halves. The diagnostic half is a property of current readers. Hold the token count and the embedding similarity to the question fixed, and compare distractors that place the question entities and a plausible answer in an unbound configuration against topical distractors that lack this configuration. The lures should produce a markedly larger accuracy drop, and the model's wrong answers should disproportionately be the planted candidate. The method half is that binding, not relevance, is the right criterion for pruning evidence. A filter built on binding should close most of that gap. A relevance-based filter of equal cost, such as a cross-encoder reranker or a query-likelihood score, should not.
What the Two Literatures Supply
The co-occurrence literature supplies the mechanism, and its results set our expectations. From N-Grams to Collocations: An Evaluation of Xtract added a syntactic parse check to statistical collocation finding and raised precision from 40% to 80% at 94% recall. A Study on Effectiveness of Syntactic Relationship in Dependence Retrieval Model showed that dependency-linked term pairs beat proximity-linked pairs for ranking, by 8.8–17.0% average precision on verbose queries. Rhetoric Map of an Answer to Compound Queries required query terms to lie along connected nucleus–satellite discourse chains, reaching 78.0% versus 66.2% for Bing on TREC. It also degraded on symmetric relations such as joint and sequence, which constrains us: a discourse-only binding test will be fragile, so we pair it with dependency paths.
Ranking Gene-Drug Relationships in Biomedical Literature Using Latent Dirichlet Allocation is a second constraint. Moving from abstract-level to clause-level co-occurrence raised precision from 48.6% to 64.5% but lost roughly half the true relations. Strict binding has a recall cost we must measure, not assume away. Fraudulent Support Telephone Number Identification Based on Co-Occurrence Information on the Web and Is It the Right Answer? Exploiting Web Redundancy for Answer Validation give two further motivations. The first shows that layout co-location manufactures strong spurious associations. The second shows that asymmetric conditional scores validate answers better than symmetric association. None of these works involves an LLM reader. Our contribution is to ask whether their diagnosis transfers to one.
The distraction literature supplies the setting and the competitors. Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering (DAS) penalizes answers using the query's likelihood given each document and adds an abstention instruction. CompAct compresses retrieved documents iteratively. Aligning Extraction and Generation for Robust Retrieval-Augmented Generation (Ext2Gen) trains the reader with preference optimization to extract relevant sentences first. Superposition Prompting prunes documents processed on parallel paths. Beyond RAG vs. Long-Context (LDAR) learns how much retrieved context to admit. All of them decide what to keep by relevance or saliency. None tests against distractors that are built to be relevant-looking but unbound. The file itself flags the Ext2Gen evaluation as using randomly sampled irrelevant chunks.
The closest existing work is structural. HugRAG names "local spurious noise" from semantic similarity. It addresses this with LLM-verified causal links between knowledge-graph communities and a prompt telling the reader to ignore coincidental correlations. Its work happens at the graph level, and the reader-side defence is only an instruction. Disco-RAG builds discourse trees for each chunk and appends them, together with a plan, to the prompt, gaining 12.74 points of LLM Score on Loong. We differ from both in three ways. First, we use structure to delete unbound evidence rather than to add more tokens. The file notes a live tension here: DAS and Less is More show that extra prompt text increases distraction. Second, we work at the sentence level inside passages. Third, we first establish, with controlled data, that unbound co-occurrence is the operative failure.
Research Program
The first component is a controlled lure benchmark built on HotpotQA and MuSiQue. Both datasets annotate supporting sentences, which lets us define lures without new annotation of relations. HotpotQA has a distractor setting with sentence-level supporting facts. MuSiQue was explicitly built to defeat disconnected shortcuts, so its multi-hop questions stress the recall cost of strict binding.
We will build three distractor conditions for each question. Each will be matched to the gold context on token count and on dense-retriever similarity to the question, and will be swapped in for the same number of non-gold passages. The random condition uses unrelated passages. The topical condition uses passages similar to the question that do not contain the question's entities alongside an answer-typed span. The lure condition uses passages containing the question entities and an answer-typed span that is not a supporting fact.
Lures will come from two sources. Mined lures are naturally occurring non-supporting sentences from the corpus that contain the entities. Constructed lures are LLM rewrites of a supporting sentence that keep the entities and introduce a same-type candidate in an unbound position, for example in a parenthetical, a separate clause, or a list or boilerplate fragment. We will validate a human-checked sample of constructed lures for being genuinely unbound and non-entailing. We will also vary the lure dose, from one lure up to several, to obtain a dose–response curve.
The second component is the binding filter. Each retrieved passage is split into sentences, and each sentence is dependency-parsed and segmented into discourse units. A sentence is kept as answer-bearing evidence when two conditions hold: the question's entities and some candidate span are linked by a short dependency path, and they fall within one discourse unit or one connected nucleus–satellite chain. Sentences that mention the entities but fail the test are not deleted. They are moved into a clearly marked background block, which protects bridge sentences in multi-hop questions.
We will build a second implementation of the same criterion in which a small instruction-tuned model answers, for each sentence, whether it asserts a relation between the named entities. This separates the claim that binding is the right criterion from the claim that parsers are the right tool. As a tie-breaker, the filter adds the asymmetric conditional score motivated by the answer-validation work: the likelihood of the question given the sentence, normalized against the likelihood of the sentence given the question. The filtered context then goes to an unmodified reader.
Baselines
The naive reference point is the reader given the full matched context with no filtering. It tells us how large the lure effect is in the first place. Every method is run with the same retrieved sets, the same reader, and the same decoding settings.
The most directly competitive method on this question is Ext2Gen, because it already trains the reader to extract the right sentences before answering. If any existing method learns to reject unbound sentences, it is this one. The file reports, for Llama3.1-8b-instruct, an average score of 0.557 for Ext2Gen-R2 against 0.473 for the default model, across test sets that include HotpotQA (Aligning Extraction and Generation for Robust Retrieval-Augmented Generation). We will run Ext2Gen-style training on the same base model with the published recipe, using the same base model and data for any training of our own components. We will evaluate it on all three distractor conditions. Beating it means a higher answer F1 and a lower lure rate on the lure condition at matched context length, without a statistically significant loss on the random and topical conditions.
The strongest training-free competitor is DAS, which scores documents by query likelihood. Its reported results are relative: an average 64% Exact Match gain over naive zero-shot readers, and 231% on SQuAD with 100 DPR documents (Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering). We will run it with the same reader. For the compression family we run CompAct, which the file reports as 7.0 F1 above RECOMP on HotpotQA (CompAct). We also run a cross-encoder reranker that keeps the top sentences up to the same token budget our filter produces.
For absolute grounding on HotpotQA's distractor setting, the original baseline was 58.28 answer F1 against a human score of 91.40 (HotpotQA). We found no reported result on Lacuna for any method on co-occurrence-controlled distractors, so every number in the lure condition will be new.
The key ablation swaps the binding test for a relevance test of identical cost and identical token budget. It uses the same small model, now asked "is this sentence relevant to the question?" instead of "does this sentence relate these entities?" If the ablated version matches the full filter, the binding criterion is not doing the work.
Why not just…?
Why not just use a strong cross-encoder reranker? A reranker is trained to score relevance, and a lure is relevant by construction: it contains the question's entities and an answer of the right type. HugRAG attributes its "precision gap" to exactly this kind of similarity-driven retrieval (HugRAG), but that is an argument, not a measurement on lures. The token-budget-matched reranker baseline settles the question directly. If the reranker's lure rate is as low as the binding filter's, the method half of our claim fails, and the paper becomes a benchmark paper.
Why not just fine-tune the reader to extract evidence, as Ext2Gen does? This is the most serious threat. Preference training on the right data might teach binding implicitly. Its training negatives, however, were random irrelevant chunks, so there is no evidence yet either way. We will test the published-recipe model on lures. We will also train a variant whose rejected examples include mined lures. If that variant matches our filter, we will report that the contribution is the lure data rather than the filter, and that our filter remains the option for black-box readers that cannot be fine-tuned.
Why not just tell the model to ignore coincidental mentions? HugRAG's spurious-aware prompting does exactly this. The evidence on self-filtering is discouraging: SKR found direct prompting the weakest way to elicit what a model knows (Self-Knowledge Guided Retrieval Augmentation), and DAS needed score adjustment on top of its abstention instruction. We will include a spurious-aware prompt as a baseline on the same reader.
Why not just ask a small LLM to judge binding and skip the parser? That may well work as well, which is why it is a planned implementation rather than a dismissed alternative. The parser version matters only if it is close in accuracy at far lower cost. We will report both versions with latency, and treat the criterion, not the tool, as the contribution.
Proposed Experiments
The first decisive check is the diagnostic one. On HotpotQA and MuSiQue, with Llama3.1-8b-instruct and one black-box API reader, we compare answer F1, supporting-fact F1, and lure rate across the random, topical, and lure conditions at matched length and similarity. Lure rate is the fraction of errors that equal the planted candidate. If lures do not cause a clearly larger drop than topical distractors, with errors concentrated on the planted candidate, the premise is wrong and we stop there.
The second check is the main comparison. The binding filter, in both implementations, is compared against the baselines above, reporting F1, lure rate, and tokens passed to the reader. Success means a lower lure rate and higher F1 on the lure condition while staying within noise of the best baseline on the standard conditions.
The third check is the recall cost that the gene–drug work warns about. On MuSiQue we break down accuracy by hop count and measure how often bridge sentences fail the binding test. We then test whether the background block recovers them, compared against hard deletion.
The fourth check is the Disco-RAG tension. On a subset, we compare appending discourse trees to the prompt against using the same trees only to filter, at equal reader budget. This tells us whether structure helps more as a filter than as extra text.
References
- From N-Grams to Collocations: An Evaluation of Xtract
- A Study on Effectiveness of Syntactic Relationship in Dependence Retrieval Model
- Rhetoric Map of an Answer to Compound Queries
- Ranking Gene-Drug Relationships in Biomedical Literature Using Latent Dirichlet Allocation
- Fraudulent Support Telephone Number Identification Based on Co-Occurrence Information on the Web
- Is It the Right Answer? Exploiting Web Redundancy for Answer Validation
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
- CompAct: Compressing Retrieved Documents Actively for Question Answering
- Aligning Extraction and Generation for Robust Retrieval-Augmented Generation
- Superposition Prompting: Improving and Accelerating Retrieval-Augmented Generation
- Beyond RAG vs. Long-Context: Learning Distraction-Aware Retrieval for Efficient Knowledge Grounding
- HugRAG: Hierarchical Causal Knowledge Graph Design for RAG
- Disco-RAG: Discourse-Aware Retrieval-Augmented Generation
- Less is More: Summary of Long Instructions is Better for Program Synthesis
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
- ♫ MuSiQue: Multihop Questions via Single-hop Question Composition
- Self-Knowledge Guided Retrieval Augmentation for Large Language Models