Retrieval-augmented generation (RAG) means handing a language model a few passages fetched from a document collection before it answers. It fails in a specific and puzzling way. Passages that look related to the question but do not contain the answer badly hurt accuracy. On Natural Questions, adding 18 such "distractors" to the correct passage drops Llama 2 7B accuracy from 56.42% to 37.81%. Yet adding completely random passages to a realistic retrieved set raises accuracy from 18.66% to 25.30% (Rethinking Relevance). We propose a simple account of this paradox and a fix built on it. A question imposes a small set of checkable constraints: what kind of thing the answer is, which entity it concerns, which time period, which relation. A distractor is harmful to the extent that it is a near miss: it satisfies most of those constraints and offers a well-typed candidate answer that breaks just one of them. Random text satisfies none, so the model ignores it.
The fix follows from the account. Before generation, extract from each passage the fact it offers as an answer (its candidate "binding"), write the question's constraints as a small typed formula, and use an exact solver to test whether each binding is consistent with them. The check is in the spirit of classical rule-base verification. A passage is masked only when the solver proves a violation. Anything the extractor cannot parse is kept, so recall is protected by design. The two claims are that harm grows with how many constraints a non-answer passage satisfies, and that removing only proven near misses recovers most of the lost accuracy while keeping the benefit of harmless noise.
Core Claim
Existing defenses against distractors score relevance: similarity, likelihood of the query given the document (DAS), or learned preference for helpful sentences (Ext2Gen). Near misses are dangerous precisely because they are highly relevant, so relevance is the wrong axis. The right axis is constraint satisfaction. Write $C(q)$ for the conjunction of typed constraints extracted from question $q$, and $B(p)$ for the binding extracted from passage $p$. The gate is
$$\text{keep}(p) \iff \text{SAT}\big(C(q) \wedge B(p) \wedge K\big) \ \text{or}\ B(p)=\varnothing,$$
where $K$ is a small set of background facts: sort hierarchies, date arithmetic, and the passages' own mutual consistency. $B(p)=\varnothing$ means extraction failed. The diagnostic prediction concerns a non-gold passage that satisfies $k$ of the $m$ constraints in $C(q)$. Its marginal damage to accuracy should rise with $k/m$ and peak at $k=m-1$. Random passages sit at $k\approx 0$, which is why they are harmless.
The design lifts two ideas from the classical literature on proving rule systems consistent. First, from discourse consistency via many-sorted logic, which infers the types of entities in a text and proves inconsistency when the allowed types have no overlap, we take sort inference as the cheapest and most reliable check. Second, from CHECK and SVEPOA, which statically verify expert-system rules by reducing them to satisfiability and linear inequalities, we take the principle of proving conflicts before running the system. Neither line ever faced uncurated text. We sidestep their formalization bottleneck by formalizing only the question's constraints and one binding per passage, never the whole passage.
Relation to Prior Work
The closest systems check consistency with neural judgments rather than proofs. SeCon-RAG extracts entity–intent–relation graphs and filters passages for query, corpus, and model consistency using LLM evaluations and graph heuristics. It targets poisoning, and its "model consistency" axis trusts parametric memory. We differ in two ways: we check against the question's explicit constraints with an exact solver, and we never delete on an unproven judgment. Logical Consistency of LLMs in Fact-Checking formalizes negation and De Morgan consistency and fine-tunes models to respect it. It assumes a clean knowledge-graph context, whereas we study precisely the noisy context it sets aside. ORAG uses an ontology as a retrieval scaffold for entity typing. We use types as checkable constraints for open-domain QA. NCP-Bench audits agents against a "fact ledger" and motivates our ledger-style binding representation, but its auditor is itself an LLM.
Several works motivate and constrain the design. Two findings rule out delegating the gate to the generator's own judgment. HaluEval shows ChatGPT recognizes hallucinations at only 58.53% accuracy in summarization. CoV-RAG uses exactly such self-scoring in a re-retrieval loop. CLEAR and TruthProbe intervene inside the model's activations: CLEAR forces the model to attend to conflicting external text, and TruthProbe amplifies the attention heads that track truthfulness. Both are complementary to an external gate, but they leave no auditable reason for a decision, and CLEAR amplifies near misses if a near miss is what conflicts. RGB supplies the evidence that noise degrades multi-document integration from 60–67% to 34%. This is where joint checks across passages should matter most.
Research Program
The work has three parts. First, a near-miss corpus. For NQ-open and HotpotQA questions, an LLM extracts $C(q)$ as typed constraints: answer sort, anchor entity, temporal window, relation, and numeric comparison. Top-ranked non-gold passages from Contriever and BM25 are then labeled by how many constraints they satisfy. A 500-question sample is human-verified to measure extraction precision and recall. Second, the diagnostic. Following the injection protocol of Rethinking Relevance, we inject passages stratified by $k/m$ and measure accuracy loss per stratum, alongside random-noise controls. Third, the gate. Bindings are compiled to a finite-domain satisfiability problem over sorts, dates, numbers, and equalities; this is decidable and fast at this size. We run the gate per passage and then jointly across the kept passages, so that for multi-hop questions a pair of bindings that cannot both hold is flagged. Masked passages are either dropped or replaced by random filler, which tests whether the regularization effect of noise can be deliberately kept. Every decision is logged with the violated constraint.
Baselines
The naive reference is the plain reader: the same generator on the same top-$k$ passages with no filtering, together with the oracle reader given only gold passages. The gap between them defines the recoverable accuracy. Beating the naive reader is necessary but uninteresting.
The strongest existing method for distractor robustness is Ext2Gen. It trains the model with preference optimization to extract relevant sentences before answering, using up to 25 injected distractors. It reports an average QA score of 0.557 against 0.473 for default Llama3.1-8b-instruct. It is the strongest competitor because it was trained on exactly this failure. We will use the same Llama3.1-8b generator, apply our gate in front of both the default and the Ext2Gen-style model, and evaluate on identical injected sets. Winning means a statistically significant gain on the near-miss strata ($k \ge m-1$) with no loss on gold-only inputs. There is no reported result on Lacuna for Ext2Gen under the stratified near-miss protocol, so we will measure it ourselves.
For conflicts and poisoning, the reference is SeCon-RAG, which reports 83.6% accuracy with 2.4% attack success rate under 100% poisoning on HotpotQA with GPT-4o. We will compare using the same generator and poisoning setup where it can be reproduced, and report latency, since SeCon-RAG's multi-call LLM pipeline is expensive. DAS is a training-free baseline, reported as a 64% average exact-match gain over naive zero-shot readers. We run it at the same retrieval depth. A cross-encoder reranker with a tuned cutoff is the obvious engineering baseline. We give it the same tuning budget on a shared development split as our extraction prompts receive.
The key ablation isolates the solver. The same LLM that extracts constraints is instead asked to judge, holistically and then constraint-by-constraint, whether each passage satisfies the question. This yields three conditions: holistic LLM judge, decomposed LLM judge, and decomposed extraction plus solver. All use the same model and prompt budget. A second ablation runs the per-passage gate without the joint cross-passage check.
Why not just…?
Why not just use a stronger reranker? Rerankers are trained to score relevance, and near misses are by construction the most relevant non-answers. That is why they sit at the top of dense retrieval in the first place (Rethinking Relevance). The stratified diagnostic settles this directly. If a tuned cross-encoder already pushes $k=m-1$ passages below the cutoff at a rate matching the gate, the proposal's intervention is redundant and only the diagnostic stands.
Why not just ask the LLM whether each passage satisfies the question? This is the real threat. Our extractor is itself an LLM, and decomposed prompting alone may capture most of the gain. The evidence against self-judgment (HaluEval; the sycophantic yielding shown in NCP-Bench) concerns judging one's own outputs, not reading passages. Two things should favor the solver. Date and numeric comparisons are where LLM judges err. Joint inconsistency across passages is invisible to per-passage judging. The three-way ablation decides it. If the decomposed LLM judge matches the solver on single-hop questions, we will say so and confine the solver claim to multi-hop and numeric-constraint questions.
Why not just train the model to be robust, as Ext2Gen does? Preference training helps, but it gives no reason for rejecting a passage and must be redone for each generator. The gate is model-agnostic and composes with Ext2Gen, so the fair test is gate-plus-Ext2Gen against Ext2Gen alone. If the gate adds nothing on top, trained robustness has already absorbed the near-miss signal, which would itself confirm the diagnostic account.
Proposed Experiments
The decisive check is the damage curve. On NQ-open with Llama 2 7B-Chat and Contriever, reproducing the published 56.42% gold-only setting, we measure accuracy loss per injected passage as a function of $k/m$. The account predicts a monotone rise that peaks at $k=m-1$, with random passages near zero or slightly positive. A flat curve, where passages hurt equally regardless of how many constraints they satisfy, falsifies the account and the gate's rationale.
The second check is recovery. With 18 stratified distractors, how much of the drop to 37.81% does the gate recover? And does replacing masked passages with random filler beat dropping them, reproducing the 18.66%→25.30% noise benefit on purpose?
Third, on the RGB suite we report noise robustness, negative rejection (baseline: ChatGPT ~45%), information integration (baseline: 34% under noise), and counterfactual robustness (baseline: 57.14% error correction). The joint check should matter most on integration, and on negative rejection, where masking every binding becomes a principled reason to abstain. HotpotQA tests joint checking across hops.
Throughout, we report extraction precision on the human-audited sample, the false-mask rate on gold passages (the target is under 2%, since the gate removes only proven violations), and wall-clock overhead against SeCon-RAG. A gate that masks gold passages often indicates that extraction, not reasoning, is the bottleneck. That result is worth reporting either way, because it measures how much of open text can be checked this way.
References
- Rethinking Relevance: How Noise and Distractors Impact Retrieval-Augmented Generation
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
- Aligning Extraction and Generation for Robust Retrieval-Augmented Generation
- Discourse Consistency and Many-Sorted Logic
- Checking an Expert Systems Knowledge Base for Consistency and Completeness
- A Methodology for Systematic Verification of OPS5-Based AI Applications
- SeCon-RAG: A Two-Stage Semantic Filtering and Conflict-Free Framework for Trustworthy RAG
- Logical Consistency of Large Language Models in Fact-Checking
- ORAG: Ontology-Guided Retrieval-Augmented Generation for Theme-Specific Entity Typing
- Can LLM Agents Stick to the Script? A Benchmark for Long-Horizon Consistency in Interactive Narratives
- HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
- Retrieving, Rethinking and Revising: The Chain-of-Verification Can Improve Retrieval Augmented Generation
- Probing Latent Knowledge Conflict for Faithful Retrieval-Augmented Generation
- The Truth Stays in the Family: Enhancing Contextual Truthfulness via Inherited Heads in Model Lineages
- Benchmarking Large Language Models in Retrieval-Augmented Generation