Filtering Distractors Inside the Reader: Decode-Time Document Masking for Retrieval-Augmented Generation

Language models that answer questions from retrieved documents are easily misled by documents that look relevant but do not contain the answer. The standard fix adds another model in front of the reader that scores the documents and drops the bad ones. That extra model adds another step that must finish before generation can start, and it judges each document alone, so it tends to throw away the "bridge" document a multi-hop question needs. We propose to move the filtering inside the reader. The model reads all retrieved documents once, with the question placed first. A tiny probe on the model's own internal states then marks which documents are unhelpful, and those documents are hidden from attention for every generated token. Nothing is re-read, no second model runs in the critical path, and documents are judged after they have already been read together with each other.

The bet rests on one empirical premise we test first. Much of the damage distractors do happens while the answer is being written, as attention during generation spreads over irrelevant tokens. It is not fixed permanently during the initial read. If that premise holds, hiding documents after reading should recover most of the accuracy that removing them beforehand would give. Because the documents were read jointly, cross-document links needed for bridge questions should survive in the remaining documents' stored states. Pipelines that drop documents before the reader, or that read each document in isolation, cannot keep those links.

Motivation and Prior Work

Three findings motivate the proposal.

  • Similar-looking distractors are the dangerous kind. Rethinking Relevance shows that adding 18 high-similarity distractors dropped Llama 2 accuracy from 56.42% to 37.81%, while random Wikipedia noise did not hurt.
  • Document count matters on its own. More Documents, Same Length holds total length fixed and still sees up to 20% degradation on 2WikiMultiHopQA and 10% on MuSiQue as the document count rises from 2 to 20.
  • Extra stages cost real time. Towards Understanding Systems Trade-offs in RAG Inference measures time-to-first-token nearly doubling, from 495ms to 965ms, once retrieval and the initial read of the context (the "prefill") are added. Prefill alone accounts for 45% of that time.

Together these say that noise hurts accuracy, and that each added stage hurts latency.

Existing fixes each pay one of these costs.

  • InfoGain-RAG (InfoGain-RAG) trains a separate 335M RoBERTa filter on each document's standalone information gain, $p(y|x,d)-p(y|x)$. This supplies our labelling idea. However, it adds a sequential stage and scores documents one at a time, which undervalues a bridge document that is useless alone.
  • Superposition Prompting (Superposition Prompting) reads each document in its own parallel path and prunes paths. It is fast, but by construction it blocks cross-document attention before pruning.
  • Speculative RAG (Speculative RAG) splits documents across parallel drafters. Its authors report it is "less effective for bridge-type questions."
  • Ext2Gen (Aligning Extraction and Generation) avoids an external stage by training the reader to extract relevant sentences before answering. It still reads all noisy chunks and adds extraction tokens to the output.

The closest mechanisms come from speculative decoding, the technique where a cheap drafter proposes tokens and the large model checks several at once.

  • RAPID (RAPID) has a drafter work on a short retrieved context while the target model keeps its full 128K context in memory and stays exposed to distraction.
  • Vegas (Vegas) uses the checking step's attention to pick which stored entries a sparse drafter reads. Its goal is speed only, and it was evaluated on clean math, code and long-context data.

Neither changes what the target model itself attends to in order to remove distractors. That is our intervention.

Core Claim

A reader model can filter its own distractors at document granularity during generation. Doing so matches or beats an external pre-filter on single-hop questions, clearly beats it on bridge-type multi-hop questions, and removes the filter stage from the critical path.

Concretely, the procedure has four parts.

  1. Order the prompt. The prompt is ordered question first, then documents, so each document's internal states are computed with both the question and the earlier documents in view.
  2. Score documents from the reader's own states. A two-layer probe reads mean-pooled middle-layer states of each document and predicts a keep-or-hide decision. It costs microseconds on states that already exist.
  3. Hide the rejected documents. Their stored attention entries are masked for all subsequent decoding steps.
  4. Train the probe on leave-one-out usefulness. Standalone gain is replaced by leave-one-out usefulness measured with the same masking operation used at test time:

$$u(d) = \log p(y \mid x, D) - \log p(y \mid x, D \setminus \{d\}),$$

Here $x$ is the question, $y$ the gold answer, $D$ the retrieved set read jointly, and $D\setminus\{d\}$ means document $d$ is masked at decode time after joint reading. A bridge document that only matters in combination gets positive $u(d)$. A dangerous neighbor that pulls the answer away gets negative $u(d)$.

Research Program

The work proceeds in three stages. Each can falsify the idea on its own.

Stage 1: oracle premise study. On NQ with distractors injected in the style of Rethinking Relevance, and on MuSiQue and 2WikiMultiHopQA in the fixed-length multi-document setting of More Documents, Same Length, we compare three conditions using gold supporting-fact annotations:

  • reading everything;
  • reading only gold documents;
  • reading everything and then masking all non-gold documents at decode time.

The quantity of interest is the fraction of the gap between the first two conditions that the third closes. We run it with both prompt orders and with Llama-3.1-8B-Instruct, which has a low distraction threshold, and Qwen2.5-7B, which More Documents, Same Length reports as nearly immune to document count. If decode-time masking closes little of the gap, distraction is baked in during the initial read and the approach is dead. We would publish that as a negative result about where distraction lives.

Stage 2: probe and labels. We compute leave-one-out labels on the training splits of NQ, HotpotQA and MuSiQue using batched masked forward passes, then train the probe per model. We study which layer to read, calibration of the keep threshold, and whether the probe transfers across datasets.

Stage 3: systems integration. Masking is implemented as a per-document attention mask in a standard serving stack. We also combine it with self-speculative decoding, so drafter and checker share the masked stored states and drafting reads fewer entries. We then measure end-to-end latency at the median and 99th percentile.

Following LiteStage, which found that lowered confidence produces longer, more repetitive outputs that raise total latency, we record output length as a latency component. We test whether distraction inflates it and whether masking reverses that.

We state the cost plainly. Our method still reads every retrieved document once, so its time-to-first-token cannot beat a pre-filter that drops documents before reading. The question-first ordering also forfeits per-document stored-state reuse of the kind Superposition Prompting exploits. The claim is about the accuracy versus end-to-end latency trade-off, not first-token latency alone.

Baselines

Naive reference. The reader sees the top-20 retrieved documents concatenated, with the question at the end, and no filtering. Every method uses the same retriever output, reader weights, decoding settings and GPU.

Prompt-only mitigation. We use the negation escape hatch plus query-likelihood answer selection (DAS) from Improving Zero-shot Reader, which reports a 64% average exact-match gain over naive zero-shot readers. It is the cheapest serious competitor.

Strongest existing filter: InfoGain-RAG. We retrain its 335M reranker on the same training questions and with the same labelling compute as our probe, labelled against our reader. It then filters the same top-20 before reading. InfoGain-RAG reports a 17.9% exact-match improvement over naive RAG, and 12.5% over a BGE reranker, on NaturalQA. For MuSiQue and 2WikiMultiHopQA we found no reported InfoGain-RAG result on Lacuna.

We count as beating it any of the following:

  • statistically indistinguishable exact match on NQ at lower end-to-end latency;
  • a clear exact-match or F1 gain on the MuSiQue bridge-question subset at no worse than 1.2x its end-to-end latency.

Multi-document architectures.

  • Superposition Prompting is the prior method most comparable in speed. Superposition Prompting reports a 93x compute reduction and 43% accuracy gain on NaturalQuestions-Open, and "often the only method capable of achieving non-zero scores on the most difficult multi-hop queries" on MuSiQue.
  • Speculative RAG reports up to 12.97% accuracy gain and 50.83% latency reduction on PubHealth (Speculative RAG). We run it on MuSiQue with drafter and checker sized to match our total GPU memory.
  • Ext2Gen is trained by DPO on the same reader. Aligning Extraction and Generation reports an average score of 0.557 against 0.473 for the default Llama-3.1-8B-Instruct.

Ablations. Each isolates one ingredient:

  • Question-last ordering tests whether query-aware document states matter.
  • Standalone-gain labels instead of leave-one-out labels tests the bridge-document claim.
  • Selection by attention mass on the first answer tokens, a Vegas-style signal, tests whether the reader's own attention already points to the right documents or to dangerous neighbors.
  • Using our probe's decisions to drop documents before a second, fresh read separates the value of the selection signal from the value of joint reading.

Why not just…?

Why not just use the InfoGain-RAG filter before the reader? This is the real threat. On single-hop NQ we expect it to roughly tie us on accuracy and beat us on first-token latency, because it avoids reading dropped documents. Our argument is confined to two points. First, standalone scoring cannot credit bridge documents, and pre-reading removal destroys cross-document context. Second, its extra stage sits in the critical path and has its own latency tail. The bridge-subset comparison and the "probe decisions plus fresh read" ablation settle this. If the ablation matches our full method, joint reading adds nothing, and the contribution reduces to leave-one-out labels, which InfoGain-RAG could simply adopt.

Why not just use a model that is already robust, like Qwen2.5? More Documents, Same Length finds Qwen2.5 7B and 72B barely degrade with more documents. If the Stage 1 oracle gap is near zero for Qwen, our method only helps weaker readers, and we will say so. It would still remove decode-time attention over hidden documents, so the latency part of the claim remains testable there.

Why not just let the model's attention decide what to evict, as speculative sparse-attention methods do? Vegas shows attention from checking passes is a cheap signal of which stored entries matter for speed. A distracted model, however, attends to exactly the similar-looking documents that mislead it. The attention-mass ablation measures directly how often it keeps a distractor our probe hides.

Why not just prompt the model to ignore irrelevant context? Breaking Focus reports that such instructions proved ineffective against adaptive distractors, which still degraded models by about 45% on average. We nonetheless include the stronger DAS variant as a baseline rather than rely on that finding.

Decisive Checks

The idea survives only if three results hold.

  1. The premise holds. Oracle decode-time masking must close at least half the gap between full-context and gold-only reading for Llama-3.1-8B on distractor-injected NQ and on MuSiQue.
  2. Bridge questions improve. The leave-one-out probe must beat the retrained InfoGain filter on MuSiQue bridge questions, and it must beat its own standalone-gain ablation.
  3. Latency holds up. At matched accuracy, 99th-percentile end-to-end latency, including output length, must be no worse than the pre-filter pipeline for answers with reasoning traces.

As a secondary finding, we will report whether distraction lengthens outputs and whether masking shortens them. This would connect contextual noise to the latency paradox that LiteStage observed for layer skipping.