Role-Swapped Noise: Testing Whether Retrieval-Augmented Readers Check Who Did What to Whom

Language models that answer questions from retrieved documents are easily misled by passages that look relevant but are not. Current defenses judge relevance by overall textual fit: how likely the passage is to "produce" the question, whether it entails an answer, or whether its sentences resemble the query. None of them checks the thing a careful reader checks first, namely whether the passage actually puts the right entity in the right slot of the event. Take the query "Which company did Acme acquire?" and a retrieved passage saying "Acme was acquired by Borlan." The passage shares every content word with the query and has high overall fit. It also hands the reader a wrong answer, Borlan, sitting right next to the query's words.

We propose to build this failure into a controlled stress test and to repair it with a check borrowed from rule-based question generation. Semantic role labeling (SRL) is a parser that labels each clause's predicate and its arguments: agent (ARG0), patient (ARG1), time, location, and so on. The question-generation literature uses those labels to turn any sentence into the full set of questions it answers. Our idea is to use that machinery in two directions. First, SRL-based sentence surgery creates "role-swapped" distractors that keep the query's words but rebind them to different roles. Second, SRL-based question generation lists the questions a retrieved passage can actually answer, and the passage counts as evidence only if one of those questions matches the user's query with the same role being asked about.

Core Claim

We claim that role-swapped context is a distinct failure mode of current noise defenses. Readers equipped with document-likelihood scoring, entailment filtering, or sentence-level compression should lose substantially more accuracy on role-swapped noise than on topical or random noise of the same length and word overlap. We further claim that a role-aware relevance score closes most of this gap without hurting performance on ordinary noise. That score asks whether the passage generates the query's question with the asked-about role intact.

The claim is falsifiable in both halves. The failure mode may not exist, if strong entailment models or rerankers already handle role inversions. The repair may be unnecessary, if cheap fine-tuning on the same data does as well. The experiments are designed to detect either outcome.

Where This Sits

Several papers motivate the gap. Breaking Focus: Contextual Distraction Curse in Large Language Models shows that coherent but irrelevant context drops GPT-4o-mini on CommonsenseQA from 85.7% to 22.0%. It finds its distractions through an expensive error-guided tree search, and it never characterizes them structurally. Making Retrieval-Augmented Language Models Robust to Irrelevant Context identifies the tendency to copy surface entities from context and proposes an entailment-based back-off. The same paper reports that this back-off is "often too strict," discarding helpful passages. Our role-level check is meant to be strict about the core binding and lenient about everything else.

The closest existing method is Distraction-aware Answer Selection (DAS), from Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering. DAS already uses question generation against distraction: it reweights answers by the likelihood that a document generates the query. That likelihood is a single sequence probability with no notion of roles. We expect "Acme was acquired by Borlan" to score well at generating "Which company did Acme acquire?", and testing that expectation is our first experiment. We keep DAS's idea that a passage is relevant if it could have produced the question. We change what "produced" means: instead of a probability over the whole string, the passage must yield a structured match on predicate, filled roles, and the role being asked about.

Two papers supply the method. A Semantic Role-based Approach to Open-Domain Automatic Question Generation maps SRL slots to wh-words and reorders the remaining arguments, and it outperforms early neural generators on human ratings. We reuse its mapping to enumerate each passage's answerable questions. We also inherit its documented weakness: the parser misassigns arguments of copular verbs such as "is" and "are." Generating Adequate Distractors for Multiple-Choice Questions replaces SRL arguments with plausible neighbors, and human raters judged 96% of its distractors sufficiently distracting. We apply the same surgery to whole passages rather than answer options.

Two further papers define what we do not do. Syn-QG contributes the idea of back-translation as a fluency filter, but it is retracted, so we use none of its numbers. Evaluating Factual Consistency of Texts with Semantic Role Labeling (SRLScore) aligns seven-role fact tuples to score summaries after the fact. It averages matches across roles, so several matched peripheral roles can mask a contradiction in the agent. We use a veto instead: a mismatch in the asked role or a core role disqualifies the passage.

Research Program

The first component is a role-swap benchmark built from HotpotQA (HotpotQA), Natural Questions, and 2WikiMultiHopQA. For each question, we parse the gold supporting sentence with SRL and identify which role the answer fills. We then write three kinds of distractor sentence and embed each in its original paragraph.

A role inversion distractor exchanges the agent and patient using voice change, so the query's entity now fills the role the question asks about and a new entity fills the query's original role. A role substitution distractor keeps the structure but fills the asked role with a type-matched wrong entity, following the distractor-generation recipe. A modifier shift distractor alters time or location arguments for "when" and "where" questions.

Each distractor is paired with a length- and overlap-matched topical distractor (same entities, different predicate) and a random distractor. We run each item in two conditions: with the gold passage present, which tests whether the reader resists the lure, and with it absent, which tests whether the reader abstains. We filter the edited sentences with back-translation and check fluency on a human-rated sample. The cost is low because the surgery is rule-based and needs no search.

The second component is the role-aware relevance score. For each retrieved sentence, we enumerate its SRL-derived questions, each recorded as a tuple of predicate, filled arguments, and asked role. We parse the user query the same way. SRL on questions is less reliable, so we also generate candidate query parses with a small language model and keep the parse that best matches.

For a passage sentence $s$ and query $q$, the score is

$$R(s,q) = \max_{g \in G(s)} \; \mathbb{1}[\text{role}(g) = \text{role}(q)] \cdot \prod_{r \in \text{core}(q)} \text{sim}\big(g_r, q_r\big)$$

The symbols are as follows. $G(s)$ is the set of questions generated from $s$, and $\text{role}(\cdot)$ is the role being asked about. $\text{core}(q)$ is the set of predicate and core arguments the query fills. $\text{sim}$ is an embedding similarity between the corresponding spans. We use a product rather than an average so that a single bad core binding sends the score toward zero, which is the veto that SRLScore lacks.

We plug $R$ into the reader in exactly the place DAS uses $P(\text{query}\mid\text{document})$. Answers from passages with low $R$ are penalized, and the reader abstains when no passage passes. We also try a pruning variant: the reader sees only the matching predicate-argument frame plus its sentence, which aims at the within-sentence noise that sentence-level extractors leave behind.

The third component is analysis. We measure how often real retrieval errors are role mismatches, since a benchmark built only from synthetic edits could describe an artifact. We also measure how much the pipeline's results depend on SRL parser errors, by comparing gold-edited role labels against parsed ones.

Baselines

The naive reference point is a zero-shot instruction-tuned reader (a Llama 3.1 8B instruct model) fed the raw top-k passages with no filtering.

The prompting baseline adds an instruction to ignore irrelevant text and answer "unanswerable" when unsupported. This is the defense Breaking Focus found ineffective, and we include it to confirm that finding on role-swapped noise.

The strongest existing inference-time competitor is DAS. It is the only published method that already uses question generation against distraction, so it isolates our specific change. DAS reports an average 64% relative Exact Match gain over naive zero-shot readers across NQ, TriviaQA, WebQuestions, and SQuAD, and a 231% gain on SQuAD with DPR over 100 documents. The file gives no absolute EM values for DAS, so we re-run it ourselves: same reader, same retrieval, same k, and its own query-generation model. Beating DAS means a significant EM improvement over it on the role-swap split at matched abstention rate, with no loss on the topical and random splits.

The entailment back-off from Making Retrieval-Augmented Language Models Robust to Irrelevant Context is the most direct structural competitor, because a good entailment model might reject role inversions on its own. We run it with the same entailment checkpoint and threshold tuning budget as our score. We also report its false-rejection rate on gold passages, since its known weakness is over-pruning.

The compression baselines are the RECOMP extractive compressor (RECOMP) and Ext2Gen (Aligning Extraction and Generation for Robust Retrieval-Augmented Generation). Ext2Gen reports an average score of 0.557 versus 0.473 for the default Llama3.1-8b-instruct on its noisy-chunk benchmark. Both select whole sentences, and a role-swapped sentence looks relevant at the sentence level, so we expect both to pass it through.

We add a cross-encoder reranker tuned on the same development split.

We also train a role-swap direct preference optimization (DPO) model: the reader fine-tuned on clean versus role-swapped pairs generated by our own benchmark code, using a data budget matched to SA-RetRobust's 1,000 examples.

Two ablations isolate the key ingredient. The first removes the role-agreement indicator and scores passages by plain similarity between query and generated questions. The second keeps role agreement but replaces the product with SRLScore-style averaging.

Why not just…?

Why not just prompt the model to ignore irrelevant text? Breaking Focus reports that such prompting fails to prevent its roughly 45% average degradation. DAS's authors show that a model instructed to answer "unanswerable" still assigns 0.98 confidence to a hallucinated answer. We include this baseline to confirm that role-swapped noise behaves the same way, but we do not expect it to be competitive.

Why not just use a stronger reranker or an entailment filter? This is the most serious threat to the claim that the failure mode exists. Cross-encoders and entailment models do read word order, and they may already penalize "Acme was acquired by Borlan" for the Acme-acquires query. If they close the role-swap gap on our benchmark, the first half of our claim fails and we will report that. The file gives two reasons to doubt it. The entailment back-off is documented as over-strict, which means it achieves robustness by discarding recall. UDA finds that dense encoders sometimes underperform BM25 because they smooth over exact relations. The decisive number is accuracy on the role-swap split together with the gold-passage rejection rate, and our score must win on both.

Why not just fine-tune the reader on role-swapped pairs with DPO? Breaking Focus and Ext2Gen both show that preference training on distracted versus clean examples works. Given our generator, this may match the verifier in-distribution, and we take that possibility seriously. For this reason the benchmark generator, not the verifier, is the contribution that survives either result: it is the cheap, search-free data source such training needs. The verifier must earn its place in two ways. It must transfer to readers it was never tuned with, since compressors in RECOMP fail to transfer across model families. It must also handle role types held out from DPO training, for instance training on agent-patient swaps and testing on time shifts. If DPO also wins on both, we will recommend DPO trained on SRL-generated data and treat the verifier as a diagnostic tool.

Why not just compress abstractively? RECOMP shows that abstractive compressors hallucinate intermediate links on HotpotQA and lose to extractive selection. A summarizer that paraphrases "Acme was acquired by Borlan" may even blur the voice cue that reveals the swap.

Proposed Experiments

The first decisive check tests whether the gap exists. On the benchmark, every baseline should show a larger EM drop on role-swapped than on overlap-matched topical noise, and DAS in particular should give role-swapped passages query-generation likelihoods close to those of gold passages. If no baseline shows this gap, the project ends at a negative result on the benchmark.

The second check tests the repair. The role-aware score should recover EM on role-swapped items in both the gold-present and gold-absent (abstention) conditions. At the same time it must stay within noise of DAS on standard NQ and HotpotQA retrieval, and it must reject fewer gold passages than the entailment back-off.

The third check tests realism. We hand-annotate a few hundred baseline errors on naturally retrieved NQ and 2WikiMultiHopQA contexts and measure the share caused by passages that contain the query's predicate and entities in wrong roles. A small share would narrow the claim to adversarial robustness. We will state it that way rather than overselling practical impact.

The fourth check measures parser dependence. We compare the verifier using gold role labels from our own edits against parsed labels, and we report accuracy separately for copular versus non-copular sentences, since those are where the SRL-based question generator already needed hand-written fixes.

We report EM and F1, abstention precision and recall, gold-passage rejection rate, and per-role breakdowns throughout, with multi-hop results on HotpotQA and 2WikiMultiHopQA treated as the harder test of whether a strict veto on core roles avoids the over-pruning that limits the entailment back-off.