When a retrieval-augmented language model (a model that answers by reading documents fetched for the question) is given many documents, most of them are distractors. They are topically close, sometimes deliberately mutated copies of the true evidence. Every attention head still assigns them some weight, because softmax attention never assigns exactly zero. Adding documents raises the chance that the answer is somewhere in the context, but factual accuracy stops improving and can fall (PrismRAG). Mechanistic work shows that a small set of heads, under 5% of the total, does the actual fact lookup (Retrieval Head Mechanistically Explains Long-Context Factuality). In multi-document settings a handful of "expert heads" pick out the gold document regardless of where it sits in the prompt (Expert Heads). The other roughly 95% of heads are the ones that get misled.
This note proposes a mechanism we call reversible evidence gating. At every decoding step, the expert heads' attention over document blocks acts as a vote on which documents currently matter. Documents that lose the vote are hard-masked, meaning their attention logits are set to minus infinity, for the non-expert heads only. The expert heads keep dense access to everything. Because the key-value cache (the stored per-token representations the model attends over) is never evicted, a masked document can reopen as soon as the expert heads turn back to it. We expect this to matter most in multi-hop questions, where the second-hop document only becomes relevant after the first hop is resolved.
Core Claim
The claim has three parts, and the design isolates each one. First, distractor-induced hallucination in retrieval-augmented generation is driven largely by non-expert heads mixing in distractor content. Second, zeroing that content is more effective than continuously down-weighting it. Third, the decision about what to zero must be made by the model's own evidence heads and updated during decoding, not fixed once before generation.
Concretely, let $H_E$ be the set of expert heads identified on a calibration set. Let $a_{h,t}(j)$ be head $h$'s attention to context token $j$ at decoding step $t$, and let $D$ be a document block. Each block gets a score, and a non-expert head's logit to any token in block $D$ is set to $-\infty$ at step $t$ whenever
$$s_t(D) = \frac{1}{|H_E|}\sum_{h \in H_E}\sum_{j \in D} a_{h,t}(j) \;<\; \tau \cdot \max_{D'} s_t(D'),$$
where $\tau \in (0,1]$ is a single relative threshold. Query tokens, system tokens, and generated tokens are never masked.
The closest prior work falls into three groups, and the proposal differs from each. The first group diagnoses without intervening. Retrieval Head Mechanistically Explains Long-Context Factuality shows that masking the top 20 retrieval heads drops Needle-in-a-Haystack accuracy from about 95% to 63.6%, while masking 20 random heads leaves it at 94.7%. Expert Heads uses attention voting only to rank evidence after the fact. We turn their diagnostic signal into a control signal, and we protect exactly the heads they show must not be damaged.
The second group intervenes softly on text models. GAME edits attention with gradient-derived biases and then regenerates a flagged chunk. COMPASS adds control-loop logit biases to steer attention toward the context as a whole. Neither zeroes anything, and neither distinguishes a gold document from a distractor inside the context.
The third group applies hard sparsity to hallucination, but only for images. MDSAM and VisFlow apply top-K sparsification and head reweighting to image tokens in vision-language models. We move the idea to text documents. We also replace their static, prefill-time head roles with gates that are recomputed at every step.
Research Program
The first stage identifies expert heads. On held-out HotpotQA and MuSiQue items with 2 gold and 8 distractor documents, we run the permutation procedure of Expert Heads on Llama-3-8B, Mistral-7B and Qwen-2.5-7B instruct models. We cross-check the resulting heads against the Retrieval Score of Retrieval Head Mechanistically Explains Long-Context Factuality and keep 5 to 10 heads per model.
The second stage implements the gate as a decoding-time logit mask. The gate is computed from attention the expert heads already produce, so it adds no forward passes. Block boundaries come from the retrieval step itself.
The third stage tests whether contamination from the prompt-encoding pass defeats decode-time masking. In a standard dense prefill, the representations of gold-document tokens have already attended to distractors before decoding begins. We therefore run every condition twice. One run uses standard dense prefill. The other uses Block-Attention's block-fine-tuned model, where documents are encoded independently. That paper measured latency and accuracy but never hallucination, so this comparison is new in itself.
The fourth stage handles the risk of discarding context too early. The known danger of token eviction, noted for Selective Attention Improves Transformer, is that something dropped early turns out to be needed later. Our design sidesteps this by masking without evicting. We log the full per-step gate trajectory, which lets us check directly whether documents are in fact reopened at hop boundaries.
Baselines
The naive reference point is unmodified dense decoding with all retrieved documents in the prompt. Every method uses the same retrieved set, the same greedy decoding, and the same models. For external context, PrismRAG reports that factuality stops improving as the number of documents grows toward 50.
The strongest existing inference-time competitor is COMPASS. It is single-pass, works on text, targets hallucination, and was evaluated on HotpotQA, XSum, HaluEval and RAGTruth. Its reported results are absolute hallucination-rate reductions of 2.8% to 5.8% across models and benchmarks, and a 22.4% cut in RAGTruth span density for LLaMA-2-13B. We will rerun COMPASS on our exact document sets and models, including LLaMA-2-13B on RAGTruth, so that we can check its published number. Its classifier and controller gains get the same tuning budget as our single $\tau$. Beating COMPASS means a lower hallucination rate and span density than it achieves, with non-overlapping bootstrap intervals, at comparable tokens per second.
The second strong competitor is static evidence filtering. We rank documents once with Expert Head Voting, keep the top-$k$, and regenerate from a pruned prompt. Expert Heads reports Precision@2 of 90.72% on HotpotQA for this ranking, against 75.23% for the BGE retriever. We also run the same pipeline with BGE reranking. The comparison is fair because both pipelines use identical head sets and the same $k$ selected on a development split. Beating static filtering requires a gain on multi-hop answer accuracy and hallucination, not just an efficiency gain.
The third competitor is GAME. On XSum it raised the Llama2-7B non-hallucination rate from 49% to 59% at 2.69 seconds per sample. We run it on XSum and RAGTruth summarization.
Three ablations isolate the key ingredients. The first is soft versus hard: the gate is unchanged, but masked documents receive a finite logit penalty of the same size COMPASS injects, instead of minus infinity. The second is who gets masked: we mask all heads, including expert heads, instead of non-expert heads only. The third is who decides: the gate is computed from randomly chosen heads matched by layer. If the random-gatekeeper version works equally well, the mechanistic story is wrong.
Why not just…?
Why not just rerank once and re-prompt with the top documents? This is the most serious threat, and we treat it as one. Under Block-Attention prefill, masking a document is nearly equivalent to deleting it from the prompt, so on single-hop questions we expect static filtering to match us. In that case our contribution there is only avoiding a second prefill. The proposal stands or falls on multi-hop data (MuSiQue, 2WikiMultiHopQA) and long summaries. There, the gate trajectory should show second-hop documents being reopened mid-generation, and dynamic gating should beat the best static top-$k$. If it does not, we will report that the effect is purely an efficiency effect.
Why not just fine-tune for distractor resilience? PrismRAG gains 5.4% average factuality over Llama-3.1-70b-instruct by training on synthetic mutated distractors. That approach requires building a synthetic corpus and updating the full model, and it leaves attention dense. The two approaches are complementary rather than competing, so we test whether the gate adds gains on top of a distractor-fine-tuned model. If it adds nothing there, the finding would be that fine-tuning already teaches the non-expert heads to ignore distractors.
Why not just steer attention softly toward context? COMPASS and GAME do this, but they push attention toward the context as a whole, and the whole context includes the distractors. The soft-versus-hard ablation answers this question directly with the gate held fixed. Prior denoising work in recommendation found that the advantage of hard masking over dense attention widens as injected noise grows (Probabilistic Masked Attention Networks). We therefore predict the hard-versus-soft gap will widen as the number of distractors increases.
Why not just sample several answers and check consistency? Sampling-based detection adds 400% to 800% latency (RAUQ), and multi-candidate verification took 18.93 seconds per sample in the GAME evaluation. It also only detects hallucinations rather than preventing them. We include it only as a reference on cost versus accuracy.
Proposed Experiments
The main testbed uses HotpotQA, 2WikiMultiHopQA and MuSiQue with 5, 10, 20 and 50 retrieved documents. Distractors include hard mutated versions of the gold documents, generated in the style of PrismRAG. We report answer exact match and F1, and a hallucination rate defined as answers not supported by any gold document. For long-form faithfulness we use RAGTruth with span-level hallucination density and XSum with the GAME non-hallucination rate. Needle-in-a-Haystack serves as a check that retrieval ability is preserved. HaluEval QA is included for comparability with COMPASS.
Four checks decide whether the claim holds. First, the hard gate must beat the soft gate at matched gating, and the gap must grow with distractor count. Second, dynamic gating must beat static Expert-Head filtering on multi-hop questions. Third, expert gatekeepers must beat random gatekeepers. Fourth, the dense-prefill versus Block-Attention comparison will show whether contamination during prompt encoding limits what decode-time masking can recover.
Every run records tokens per second and time to first token alongside accuracy. The method must stay single-pass, and it should approach the throughput of plain decoding. The same logs let us test the Expert Heads observation that dispersed expert-head attention precedes hallucination. We will check whether steps where every document score is low predict the remaining errors. If they do, a natural follow-up is to abstain in those cases instead of masking.
References
- PrismRAG: Boosting RAG Factuality with Distractor Resilience and Strategized Reasoning
- Retrieval Head Mechanistically Explains Long-Context Factuality
- Expert Heads: Robust Evidence Identification for Large Language Models
- Gradient-guided Attention Map Editing: Towards Efficient Contextual Hallucination Mitigation
- COMPASS: Context-Modulated PID Attention Steering System for Hallucination Mitigation
- MDSAM:Memory-Driven Sparse Attention Matrix for LVLMs Hallucination Mitigation
- Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation
- Block-Attention for Efficient RAG
- Selective Attention Improves Transformer
- Probabilistic Masked Attention Networks for Explainable Sequential Recommendation
- Efficient Hallucination Detection for LLMs Using Uncertainty-Aware Attention Heads