Passage-Level Attention Dropout for Distractor-Robust Language Models

Language models given retrieved documents or long inputs are easily derailed by irrelevant text. They copy a plausible entity from a wrong passage, lose the one sentence that matters in the middle of a long input, and often get worse as more context is added. Dropout addresses a structurally similar problem in ordinary networks. Dropout means randomly switching off parts of a network during training so that no unit can depend on one specific partner. We propose to apply that idea at the level of whole passages. During fine-tuning, the question and answer tokens are randomly blocked from attending to entire retrieved passages, while the input text and its positions stay unchanged. The model therefore cannot learn to depend on whichever span is easiest to copy, and it trains on exactly the full-length, full-layout inputs it will see at test time.

The method needs no relevance labels, no external retriever or entailment filter, and no extra computation at inference. It also tests a tension in the dropout literature. Dropout is now known to hurt single-epoch language model pretraining, yet it helps networks adapt to new tasks without losing old ones. The open question is whether a structured, post-training form of dropout buys robustness to distracting context without the knowledge costs found in pretraining. The study is set up so that a negative answer, namely that this is just data augmentation in disguise, would be visible and publishable.

Core Claim

Fine-tuning a reader on retrieval-style inputs lets the answer co-adapt to particular context spans. "Co-adapt" here means the model learns to depend on one specific span to produce its answer, in the same way dropout's original motivation describes neurons correcting each other's errors instead of learning independent features. This dependence produces the "copying trap" reported by Making Retrieval-Augmented Language Models Robust to Irrelevant Context and by Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering.

Our claim is that randomly severing the answer's attention access to whole passages during fine-tuning fixes this better than the input-side alternatives. Those alternatives delete passages, inject distractors, or mask irrelevant chunks, and all of them change the length and positional layout of the training inputs. The concrete prediction concerns how accuracy changes as distractors are added. Under passage-level attention dropout, accuracy should fall more gently than under input-level passage removal given identical exposure. It should also fall less steeply than under the inverted-U accuracy curve reported by LaRA and LDAR, in which accuracy first rises and then drops as context grows. Closed-book knowledge should be unchanged.

Relation to Prior Work

Three existing methods come closest, and each differs from ours in a specific way.

Focused Learning replaces chunks that an external retriever judges irrelevant with a <mask/> token. It adds a contrastive loss that aligns the model's internal representation of noisy and clean contexts. That masking is deterministic, depends on retriever labels, and alters the input. The authors also report that positional biases persist. Our masking is random, needs no labels, and leaves the input untouched.

LKG-RALM finds that a small set of "middle-focused" attention heads, which specialize in aggregating retrieved context, are where distraction does its damage. It suppresses those heads' attention to passages it predicts are irrelevant, at inference time. We borrow its insight that attention to passages is the failure point. We act during training instead, so no relevance predictor runs at test time.

Contextual Dropout shows that dropout rates can depend on the input and that ambiguous inputs end up with higher drop rates. It was tested only on vision and visual question answering classifiers. A learned per-passage rate in its style is a natural extension of our method, but the core proposal uses a fixed rate so that the mechanism itself is tested cleanly.

Two papers constrain the design. Drop Dropout on Single-Epoch Language Model Pretraining shows that dropout during pretraining raises loss and spreads factual knowledge across the network, which makes localized editing harder. We therefore restrict dropout to low-rank adapter (LoRA) fine-tuning and measure knowledge retention explicitly. An Empirical Investigation of Catastrophic Forgeting in Gradient-Based Neural Networks motivates the opposite expectation: dropout helps when a trained network is adapted to a new task.

Finally, Breaking Focus argues that distraction is a failure of ability rather than missing knowledge, because models answer correctly once distractors are removed. This suggests that a training-time intervention on how the model reads its context is the right lever.

Research Program

Setup. Each training input is split into passages: retrieved documents in open-domain QA, or fixed-length chunks for single long documents. The model is trained with ordinary next-token loss on the answer. On each example, every passage is independently "dropped" with probability $p$. A dropped passage's keys are removed from the attention mask for the question and answer positions in every layer. The dropped tokens still occupy their positions, keep their position indices, and are still read by the context tokens that follow them.

Fallback target. The target answer stays the gold answer even when the gold passage happens to be dropped. In that case the model must fall back on what it already knows, as in the fallback training of SA-RetRobust. An abstention variant, where the target becomes "unanswerable" when the gold passage is dropped, is evaluated separately and does require gold-passage labels.

Ablations of the mask. Our dropout leaves an indirect route open: information in a dropped passage can still reach the answer through the passages that come after it. To test whether that route matters, we compare against three variants:

  1. A "full block" variant, in which no later token can attend to the dropped passage, but positions are preserved.
  2. Plain input removal, in which the passage is deleted and later positions shift.
  3. Input removal combined with a consistency loss between the two views, which is closer to Focused Learning.

Comparing full block against input removal isolates the effect of preserving positions. Comparing our dropout against full block isolates the effect of the indirect route.

Mechanism analysis. We will measure the share of attention from answer tokens that lands on the gold passage, as a function of where that passage sits in the context. We will also measure the copying rate: the fraction of wrong answers that appear verbatim in some distractor. Finally, we will test whether the middle-focused heads identified by LKG-RALM change more than other heads.

Cost check. The cost side follows Drop Dropout's concern. We will measure closed-book Natural Questions accuracy before and after fine-tuning. We will also run a small knowledge-editing probe with ReFT to see whether adapter-level dropout makes facts harder to edit.

Baselines

Naive reference. The do-nothing reference is the same base model fine-tuned with identical data, adapter rank, steps, and learning-rate sweep, but with no dropout of any kind. Focused Learning reports this kind of vanilla fine-tuning at 44.6 exact match (EM) on NQd and 48.65 F1 on Qasper, using Mistral with LoRA. We use the same base model so that our reproduction can be checked against those figures.

Input-level passage dropout with matched exposure. This is the ablation that isolates our key ingredient. It draws the same random passage subsets at the same rate $p$, but deletes the passages from the input instead of masking attention. Every gradient step therefore sees the same evidence configurations, and only the train-time input layout differs. Our method beats it only if passage-level attention dropout is better on NQd and on the accuracy-versus-distractor-count curve across three seeds, by more than the spread between seeds.

Standard dropout during LoRA fine-tuning. This is ordinary attention-weight and hidden-unit dropout at matched rates swept over the same grid. It tests whether the passage structure matters at all, or whether any stochastic regularization would do.

Focused Learning. This is the strongest existing training-time method on our exact question. It uses retriever-guided masking plus a contrastive alignment loss, and it reports 48.1 EM on NQd (52.0 with re-ranking) and 59.62 F1 on Qasper with Mistral and LoRA (Focused Learning). We will rerun it with the same base model, data, and tuning budget. Beating it means matching or exceeding its EM without any retriever labels. Combining our method with it should add further gains if the two act on different failures.

Preference-tuning baseline. For the larger-scale setting, we use Ext2Gen. It applies direct preference optimization to teach the model to extract evidence before answering, and it reports an average QA score of 0.557 against 0.473 for the default Llama3.1-8b-instruct, under up to 25 distractor chunks. We compare on the same model and benchmark. Its data pipeline is far more expensive than ours, so matching it at equal fine-tuning compute counts as a success.

Inference-time pruning. AttentionRAG serves as an inference-time reference. It prunes sentences using the attention placed on an anchor token, and it reports 0.89 against 0.87 EM on TriviaQA with a 70B model. We report it alongside our results but do not count it as a direct competitor, because it adds a second forward pass.

Why not just…?

Why not just fine-tune on contexts with injected distractors? This is the serious threat. SA-RetRobust shows that around a thousand mixed gold-and-noise examples already make a 13B model beat a 70B model under noisy retrieval. Random passage dropout could turn out to be the same augmentation in another form. The matched-exposure input-dropout baseline settles this directly. If it matches our method, the honest conclusion is that the attention-level form adds nothing, and the method reduces to a label-free augmentation recipe. We would report it as such. Our method's specific advantage, if it exists, should appear in two places: on inputs longer than those seen in training, and in the positional profile of attention to the gold passage, since only input removal shifts positions.

Why not just use ordinary dropout in the fine-tuning run? Unit-level dropout does not distinguish evidence from noise and cannot cut the answer's direct copying path to a specific span. Drop Dropout also shows that unstructured dropout can disperse knowledge. We include it at matched rates. If it closes the gap to our method, the passage structure is unnecessary.

Why not just tell the model to ignore irrelevant text? Breaking Focus reports that instructions to ignore irrelevant information fail against semantically coherent distractors, with GPT-4o-mini falling from 85.7% to 22.0% on CommonsenseQA. We still include an instruction-only arm on Qasper and NQd, because DAS shows that such prompting helps against easier noise.

Why not just filter the context before the model reads it? NLI back-off discards partially relevant passages that multi-hop questions need (Yoran et al.). LDAR chooses a band of passages but cannot fix distraction inside a document. Pruning adds latency. These approaches complement ours rather than substitute for it, and 2WikiMQA and StrategyQA, where filters are known to hurt, will show whether our trained reader handles partially relevant passages better.

Proposed Experiments

The main experiments fine-tune Mistral-7B with LoRA on the Focused Learning training mixture and evaluate on Qasper, QuALITY, and NQd. They are repeated with Llama3.1-8b-instruct on the Ext2Gen benchmark, 2WikiMQA, StrategyQA, and the 32k-to-128k splits of LaRA. The dropout rate $p$ is swept over a small grid using held-out NQd development data.

We also build distractor-scaling curves on NQ by holding the gold passage fixed and adding from zero to 25 distractors at random positions. The robustness score is the area under each curve, so it captures both the peak and the decline of the inverted-U.

For adversarial distractors, we regenerate tree-search-style coherent distractors for CommonsenseQA following the procedure of Breaking Focus. No Lacuna result exists for open 7–8B models under that attack, so all arms are compared against our own reproduction.

Three checks are decisive:

  1. Passage-level attention dropout must beat matched-exposure input dropout on NQd EM and on the area under the distractor curve. If it does not, the central claim fails.
  2. It must reduce the copying rate relative to vanilla fine-tuning while closed-book NQ accuracy stays within seed noise of the base model. This tests whether post-training dropout escapes the knowledge cost seen in pretraining.
  3. It must reach at least Focused Learning's reproduced NQd and Qasper scores without retriever labels. Its benefit should grow with context length beyond the training lengths, which is the signature that preserving positions is the ingredient that matters.