Role Frames as Noise Filters: Learning Distraction-Robust Reading from Few Examples

Language models reading retrieved documents or word problems get confused by sentences that look relevant but are not. The worst offenders mention the same people and the same kind of quantity as the question, yet describe a different relationship, for example "Mary's brother has 5 apples" when the question asks about Mary's apples. Current fixes need a lot of training data or a lot of compute. Some fine-tune the model on large sets of noisy examples. Others filter whole documents. Others sample many answers and vote. None of these tells the model which relationship the question asks about.

We propose teaching the model to read in terms of role frames. A role frame is the "who did what to whom" structure of a sentence: its main verb, the roles attached to it (agent, possessor, time, quantity), and which entity fills each role. The model first writes the frame the question needs, including an open slot for the answer. It then extracts only the context sentences whose frames can fill that slot, and answers from those. The claim is that this explicit structure makes robustness to distractors cheap to learn. With a few hundred training examples, the model should get the robustness that flat evidence extraction needs much more preference data to reach. The gain should be largest on distractors that share entities with the question but play a different role.

Core Claim

Distracting sentences that share entities and topic with the question differ from the evidence mainly in structure. The entities match; the role bindings do not. Structural priors are known to make learning cheaper when surface cues are misleading. Two examples: dependency graphs focus cross-domain sentiment transfer on modifier–noun relations instead of domain-specific words (Graph Adaptive Semantic Transfer), and subject–verb–object alignment beats token-level alignment trained on over 30 times more data (Structural Contrastive Pretraining). Both were tested only on clean inputs.

We claim that an intermediate representation matching question frames to evidence frames gives three things:

  1. A steeper learning curve for noise robustness than flat extraction trained on the same examples.
  2. A gain concentrated on the "role name overlap" distractors that Large Language Models Can Be Easily Distracted by Irrelevant Context identifies as most damaging.
  3. A natural way to keep multi-hop evidence. A document that looks irrelevant alone is kept if its frame fills an argument that another frame needs.

Relation to Prior Work

Evidence for the gap. Two papers motivate the proposal:

  • The GSM-IC study (Shi et al.) injects distractor sentences into grade-school math problems. It finds that in-topic distractors with overlapping role names drive macro accuracy below 30%, and that "ignore irrelevant information" instructions help only modestly.
  • Making Retrieval-Augmented Language Models Robust to Irrelevant Context shows that retrieval can hurt multi-hop question answering, and that failing models copy entities from distractors.

Both diagnose the problem. Neither gives the model a structural test for whether a sentence answers the question's relationship.

Methods we compete with. Two approaches define what we must beat:

  • SA-RetRobust (Yoran et al.) fine-tunes on 1,000 synthetic examples in the Self-Ask format (the model writes intermediate sub-questions and answers). Its structure is a prompt format with no representation of roles.
  • Ext2Gen (Aligning Extraction and Generation for Robust Retrieval-Augmented Generation) trains a model with Direct Preference Optimization to extract relevant sentences before answering. Direct Preference Optimization, or DPO, trains a model to prefer one output over another using paired examples. Ext2Gen beats external compressors, but it extracts flat sentences and needs pairwise preference data.

We keep Ext2Gen's key insight that the generator itself should extract. We change what is extracted: role frames bound to the question's open slot, rather than raw sentences.

Methods we reuse. We take the decomposition idea from SGSH. SGSH trains a small model to predict a grammatical skeleton that then guides a large model, and it needs only 10% of the training data. SGSH works on clean knowledge-base triples. We apply the same idea to noisy text, with a richer skeleton.

Limits we must respect. Three results constrain the design:

  • Structure pays off mostly on structured prediction, not broad understanding. Distilling syntax into BERT (Syntactic Structure Distillation Pretraining) helps parsing and semantic role labeling but gives mixed results on the GLUE benchmark.
  • Parser errors cascade, as SGSH and StructSP (StructSP) both concede.
  • Scoring documents independently can prune intermediate multi-hop evidence. Superposition Prompting does exactly this: it processes each document on its own branch and prunes low-scoring branches.

We respond to these limits in three ways. We target question answering, where the answer is fundamentally a role binding. We generate frames with the model itself and treat an external parser only as an optional supervision source. We match frames jointly across documents rather than per document.

Research Program

The representation. A question frame is a short structured string. It names a predicate, its typed roles, the entities bound to them, and one role marked as the answer slot. An example is own(possessor=Mary, theme=apples, quantity=?, time=end). Multi-hop questions become chains of frames, where the answer slot of one frame is an argument of the next.

For each candidate context sentence, the model writes its frame. It marks the sentence as supporting evidence only if the frame is compatible with a question frame. Compatible means the same predicate class, the same entity in every bound role, and a filler for the open slot or for a chained argument. The model then answers from the matched frames alone.

Training. We fine-tune Llama-3.1-8B-instruct with supervised learning. We use this model because it is Ext2Gen's reader, which makes the comparison direct. Each training sequence contains the question, a noisy context, the question frame(s), the matched evidence frames, and the answer.

We build training data at N = 100, 300, 1,000 and 3,000 examples. Contexts mix gold passages with three kinds of distractor:

  • random distractors;
  • topical distractors;
  • role-swapped distractors, built by rebinding an entity of a gold sentence to a different role, or by moving a quantity onto a relative of the protagonist, as GSM-IC does.

We produce gold frames in two ways, which we compare against each other. One is a teacher LLM prompted with a small, fixed frame inventory. The other is an off-the-shelf semantic role labeler, a tool that tags each verb's arguments with their roles.

Analyses beyond accuracy. We measure three further quantities:

  • the rate at which the model copies an entity that appears only in a distractor;
  • slot-level precision and recall of evidence selection against gold supporting facts where they exist (for example, HotpotQA supporting sentences);
  • robustness to deliberately corrupted frames, where we perturb role bindings at controlled rates to measure how much frame errors cascade.

Baselines

The naive reference is the untuned Llama-3.1-8B-instruct reader given the full noisy context, alongside a no-retrieval variant. We include no-retrieval because Yoran et al. show retrieval can hurt. We also add the GSM-IC instruction "Feel free to ignore irrelevant information given in the questions" to this reader. The reported reference point for that prompt is 72.4% to 77.8% micro accuracy on code-davinci-002 from Shi et al.. We rerun the prompt on our reader rather than compare across models.

The strongest existing method is Ext2Gen-R2. On Llama-3.1-8b-instruct it reports an average QA score of 0.557 under noisy retrieval, against 0.473 for the default model, on its noisy-chunk alignment data (Ext2Gen). It is the right competitor because it already puts extraction inside the generator and beats RECOMP, CompAct and EXIT.

We run it in two ways:

  • Replication. We evaluate on Ext2Gen's own evaluation set with its released recipe, to confirm the 0.557 reproduces in our setup.
  • Matched budget. We retrain it on our N-example budgets, converting the same noisy examples into preference pairs, with an equal hyperparameter search of 8 configurations per method and equal GPU hours.

Beating Ext2Gen means one of two things. Either we match 0.557 using a small fraction of its training data (the target is N ≤ 1,000), or we exceed it at matched N. In both cases we must also show a clearly larger gain on the role-swapped distractor slice.

SA-RetRobust (Yoran et al.) is the second trained baseline. It already reaches noise robustness with 1,000 examples and a structured intermediate format. We train it on the identical 1,000 (and smaller) examples, with the same base model and budget. On 2WikiMQA the paper reports over 30 points of improvement relative to no retrieval, but the file gives no absolute accuracy, so we rely on our rerun. We also include NLI filtering from the same paper, which discards retrieved passages that do not entail the question-answer pair. It serves as the representative pre-generation filter.

Ablations. The ablation that isolates our key ingredient is flat extraction at matched supervision. The model is fine-tuned on exactly the same examples and sees the same selected evidence sentences, but writes them as plain text with no frames. A second ablation keeps the frames but removes the matching step: the model writes frames for every sentence, then answers. This tests whether the benefit comes from binding frames to the answer slot or merely from paraphrasing into frames. A third ablation matches frames per document with no chaining, which mimics independent path pruning and tests the multi-hop argument.

Why not just…?

Why not just prompt the model to ignore irrelevant information, or vote over many samples? The GSM-IC results show that instructions leave macro accuracy under 30% when distractors share role names (Shi et al.). The same study found 99.7% answer recall with 20-sample self-consistency, but recall means only that the right answer appears somewhere among the samples, and it costs twenty times the compute. We include instruction prompting in the baselines, and self-consistency at a matched budget of sampled tokens. If voting over 20 samples beats our single pass at equal tokens, we will report that.

Why not just filter passages with an entailment classifier? Yoran et al. found NLI filtering overly conservative: it discards partially relevant multi-hop evidence (Making Retrieval-Augmented Language Models Robust to Irrelevant Context). NLI also operates on whole passages. It cannot reject a single role-swapped sentence inside an otherwise relevant passage, and that is exactly our target slice. The NLI baseline and the role-swap breakdown will show whether this holds.

Why not just fine-tune on more noisy examples with flat extraction? This is the real threat. The benefit might come from any intermediate supervision, not from structure. The flat-extraction ablation at matched N settles it. If flat extraction matches frame extraction at every N and on the role-swapped slice, the structural claim is false and we will say so. We predict the gap is largest at N = 100–300 and narrows at N = 3,000. That pattern is what "structure buys data efficiency" means. A constant gap, or no gap, would argue against it.

Why not just let the model generate frames by prompting, with no training? A zero-shot frame-matching prompt is cheap. If it works, the data-efficiency story is moot. We run it as a zero-training condition. We expect it to help on single-hop questions but to fail at consistent role binding in multi-hop chains. That expectation is untested, which is why we include this condition.

Proposed Experiments

Benchmarks. - GSM-IC, broken down by distractor type (in-topic versus off-topic, role-name overlap or not); - the Ext2Gen noisy-chunk evaluation set; - HotpotQA and MuSiQue for multi-hop question answering with distractor passages; - 2WikiMQA and StrategyQA under the random and retrieved noise settings of Yoran et al.; - NaturalQuestions-Open as a single-hop control where structure should matter least.

Metrics. Accuracy or exact match and F1 as each benchmark defines them; GSM-IC macro accuracy; the distractor-entity copy rate; and evidence precision/recall.

Decisive checks. Three results would support the proposal, and each would refute it if it failed:

  1. Data efficiency. On learning curves over N, frame extraction reaches Ext2Gen's full-data score with at most a tenth of the examples.
  2. Mechanism. Most of the gain over flat extraction at matched N comes from the role-overlap distractor slice rather than random noise.
  3. Multi-hop retention. Chained frame matching retains intermediate-hop evidence that per-document matching drops, measured by supporting-fact recall on MuSiQue.

Robustness to frame errors. The corrupted-frame experiment sets how much error in frame generation the method tolerates before it falls below flat extraction. This directly addresses the cascading-error concern raised by SGSH and StructSP.