Teaching Small Models to Ignore: Distilling Relevance Extraction Instead of Rationalization

When a language model is given a question surrounded by irrelevant or misleading text, it often reasons its way into a wrong answer. Step-by-step reasoning, also called chain-of-thought (CoT), does not reliably protect against this. Under misleading context, the model writes a longer explanation that rationalizes the false premise. The standard way to give small models step-by-step reasoning is to fine-tune them on reasoning traces written by a large "teacher" model. That method has its own known failure: small students cannot absorb long teacher traces and fall into repetitive loops. These two failures compound. A teacher that reads noisy input writes exactly the long, reconciling, self-doubting traces that small students learn worst.

This note proposes a different training target for the student. The teacher still reads the noisy input. The student learns a two-part output: a short, explicit list of the facts that matter, followed by a reasoning trace that the teacher wrote on the clean version of the problem. The noise-handling work is moved into one short extraction step at the start of the trace. The remaining reasoning keeps the length and difficulty that small students can already learn. We make one empirical claim that could be wrong. As students get smaller and distractors more numerous, the extraction step should matter more than simply pairing noisy inputs with clean traces. If that scale interaction does not appear, the simpler recipe wins and we will say so.

Core Claim

We make two claims and test them separately. The first is about measurement. Standard reasoning distillation from a long-trace teacher amplifies a student's sensitivity to distracting context, and the amplification grows as the student shrinks. We quantify this with the noise penalty:

$$\Delta(m) = \mathrm{Acc}_{\text{clean}}(m) - \mathrm{Acc}_{\text{noisy}}(m),$$

where $m$ is a trained model and the two accuracies are measured on matched clean and distractor-injected versions of the same questions. No paper in our file measures $\Delta$ across a teacher–student size gap.

The second claim is about method. Consider a student trained on targets of the form "noisy input → extracted relevant facts → clean-context reasoning → answer". We predict it will have a smaller $\Delta$ than a student trained on the teacher's noisy-context traces. We also predict that the advantage of extraction over a no-extraction control will grow with distractor count and shrink with student size.

The mechanism comes from the CoT literature's view of generated tokens as working memory. Once the relevant facts are written out, later reasoning steps can attend to a short, clean list instead of the full distracted prompt. How Is LLM Reasoning Distracted by Irrelevant Context? builds the GSM-DC benchmark from controlled computation graphs and finds that distraction widens the gap between choosing the right operations and executing them correctly. Filtering noise appears to consume capacity that arithmetic needs. Extraction pays that cost once, at the start of the trace.

How This Relates to Prior Work

Three papers motivate the problem.

Thinking in a Crowd builds SciAux, 6,828 ScienceQA-derived questions with labeled helpful, irrelevant and misleading snippets. It compares thinking-on and thinking-off modes in 7–8B models. Its numbers need careful reading. With two misleading snippets, Qwen3-8B in thinking mode scores 77.03% against 74.87% without thinking, so thinking is not strictly worse. However, thinking falls from a 96.22% clean baseline, and the paper reports that thinking adds roughly 10–13 points on clean input. Under misleading context, most of that advantage disappears, and traces become longest exactly when the model tries to reconcile the misleading text. We read this as "deliberation stops paying off and gets expensive under noise," not "deliberation always hurts." The paper tests only prompting on 7–8B models, never distillation or smaller students.

MiCoTA shows that 1.5–7B students collapse into reflection loops when trained on 32B long-CoT traces. It fixes this with a 14B teacher assistant and model merging that produces mid-length traces. It evaluates only on clean math. Effective Learning for Small Reasoning Models shows standard distillation dropping Qwen2.5-0.5B-Instruct from 45.5% to 18.7% on GSM8K, while reinforcement learning (RL) stays stable, again only on clean data. Put together, these show that noisy-context traces are long and that long traces are hard for small students. Nobody has measured what happens when both apply at once.

Two papers constrain the design. Prompt-level defenses do not work: Breaking Focus and Adaptive Distraction generate semantically coherent distractions with tree search and report average drops above 45%. They find that "ignore irrelevant information" instructions give almost no protection. That rules out a prompt-only solution and supplies a hard, held-out attack for our evaluation.

Four lines of work supply technique, and each differs from our proposal in a specific way.

  • PrismRAG fine-tunes Llama-3.1-70B on synthetic hard distractors with a plan-first reasoning format. We borrow its distractor construction. We differ in two ways: our reasoning target comes from clean context rather than being written alongside the noise, and we test sub-8B students across a size gap.
  • Aligning Extraction and Generation for Robust Retrieval-Augmented Generation (Ext2Gen) trains a model to extract relevant chunks before generating an answer. It is the closest relative of our extraction step. Ext2Gen does not use a teacher, does not separate the source of the extraction (noisy view) from the source of the reasoning (clean view), and does not study student scale.
  • RADIO distills teacher rationales into the reranker that filters retrieved documents. We put the filtering inside the generator instead, which matters because rerankers leak distractors. - **Stronger Models are Not Always Stronger Teachers** introduces the Compatibility-Adjusted Reward, which discounts a teacher output by the student's loss on it. We reuse that idea as a diagnostic: does the student's loss on each trace format, measured before training, predict how small its noise penalty will be after training?

Research Program

We build training data from two sources where the relevant facts are known exactly.

  • GSM-DC. Its generator produces math word problems from computation graphs, so every distractor and every relevant fact is labeled. Distractor count can be varied from 0 to 15.
  • SciAux. We split it into training and test questions. The helpful snippet serves as the gold extraction and the irrelevant or misleading snippets as distractors.

For each training question we create four kinds of target from one teacher, which differ only in format:

  1. Direct noisy-trace. The teacher reasons over the noisy input.
  2. Clean-trace augmentation. The noisy input is paired with the teacher's trace on the clean input, with no extraction step.
  3. Extraction plus clean trace. The proposal: a gold-verified fact list, then the clean trace.
  4. Extraction plus mid-length trace. As in (3), but the clean trace is first shortened MiCoTA-style through a 14B assistant.

The teacher is a 32B long-reasoning model, with a 14B assistant as in MiCoTA. Students are Qwen2.5 at 0.5B, 1.5B, 3B and 7B. Each student–format pair gets the same number of training tokens and the same hyperparameter search.

The analysis centers on three questions:

  • Does the noise penalty $\Delta$ grow as the student shrinks under direct noisy-trace distillation, relative to each base model?
  • Does the advantage of extraction over clean-trace augmentation, $\Delta_{\text{aug}} - \Delta_{\text{extract}}$, increase with distractor count and decrease with student size?
  • Does pre-training student loss on each format predict post-training $\Delta$ better than teacher accuracy does?

Baselines

The naive reference point is each base student with no extra training, prompted to reason step by step, plus the same model with an explicit "ignore irrelevant context" instruction. The instruction version tells us whether training is needed at all. Prior work predicts it gives almost nothing (Adaptive Distraction).

The standard distillation baseline is direct noisy-trace distillation from the 32B teacher. This is what a practitioner would do by default. Prior clean-data results suggest it may fall below the base model at 0.5B, where Effective Learning for Small Reasoning Models reports GSM8K collapsing from 45.5% to 18.7% under distillation.

The strongest existing method is a combination nobody has published: Hard-IC fine-tuning in the style of GSM-DC, using MiCoTA-style mid-length traces from a 14B teacher assistant. It joins the best-documented fix for noise with the best-documented fix for the capacity gap.

  • On the capacity-gap side, MiCoTA reports a 35.6% gain over direct distillation for Qwen2.5-3B and a 3.93-point average improvement across its clean math suite.
  • On the noise side, the file has no reported number for Hard-IC training on GSM-DC or for any distilled small student on SciAux. Our only reference points are Grok-3-Beta's step accuracy of 43% with one distractor and 19% with fifteen on GSM-DC, and Qwen3-8B's 77.03% under misleading context on SciAux.

We will also run direct preference optimization (DPO) on clean-versus-distracted pairs, as in Breaking Focus, which reports gains of nearly 50% on perturbed inputs for some smaller models. Finally, we will run GRPO-style RL on distractor-injected prompts.

The key ablation is clean-trace augmentation without extraction, because it isolates the extraction step. All methods share the same training questions, distractor sampler, token budget, and number of tuning trials.

Beating the strongest baseline means two things at the 0.5B and 1.5B scales. First, a lower noise penalty on held-out GSM-DC at 10 or more distractors and on SciAux misleading conditions, with non-overlapping 95% intervals across three seeds. Second, clean GSM8K and MATH-500 accuracy within two points of that baseline, so the robustness is not bought with a specialization tax.

Why not just…?

Why not just prompt the student to ignore irrelevant text? Breaking Focus and Adaptive Distraction found that such instructions give almost no defense against coherent distractors. Small models follow instructions even less reliably. We include the prompt as a baseline, but we expect it to only confirm this.

Why not just distill from the strongest teacher on noisy inputs? This is the standard distillation baseline. MiCoTA and Stronger Models are Not Always Stronger Teachers both show that the largest teacher's outputs transfer poorly to small students. Thinking in a Crowd shows that traces grow longest under misleading context, which is the trace type those papers find least learnable. Our first claim is precisely that this baseline gets worse as students shrink, and the experiment either shows it or refutes it.

Why not just fix retrieval so noise never reaches the generator? RADIO improves NQ exact match by 19.82% through reranking. It also reports a seesaw between domains, and PrismRAG finds that factuality plateaus as more retrieved documents bring in hard distractors. GSM-DC distractors also sit inside the problem statement, where no retriever can remove them. Reranking complements our method and cannot replace it.

Why not just run RL on the small model with distracted prompts? RL is a real competitor. Effective Learning for Small Reasoning Models shows it avoids collapse at 0.5B, but its ceiling stays low: 32.4% on MATH-500, against 45.5% when distillation is followed by RL. RL also needs verifiable answers, which SciAux has but many retrieval settings lack. We run distractor-curriculum RL as a baseline and also apply it on top of extraction distillation. If RL alone matches extraction on $\Delta$, the method contribution reduces to a better starting point for RL, and we will report it that way.

Why not just pair noisy inputs with clean-context traces and skip extraction? This is the most serious threat, and it may work nearly as well. It is the ablation the whole proposal turns on. Our prediction is specific. Extraction should help most where filtering is most costly, meaning many distractors and small students. Its effect should show up as a smaller gap between operation-selection and execution accuracy on GSM-DC. If the advantage of extraction over this ablation is flat across scale and distractor count, we will conclude that the clean-trace target alone does the work and recommend the simpler recipe.

Decisive Checks

We will run four checks.

  1. Noise penalty versus student size. We plot $\Delta$ against student size for direct noisy-trace distillation, using GSM-DC at 0, 5, 10 and 15 distractors and SciAux's misleading condition. This tests the measurement claim.
  2. The interaction test. We plot $\Delta_{\text{aug}} - \Delta_{\text{extract}}$ as a function of student size and distractor count. This determines whether extraction is worth keeping.
  3. Behavior under misleading context. We measure trace length and the rate of repeated reflection phrases. If extraction works for the reason we propose, trained students' traces should not grow under misleading context the way teacher traces do.
  4. Faithfulness. At test time we corrupt one item in the student's own extracted fact list. If answers follow the corrupted list, the student is actually using its extraction rather than treating it as decoration.

For out-of-distribution robustness, we attack each trained student with tree-searched distractions generated by the student itself on CommonsenseQA and MATH-500, following Adaptive Distraction. For reference, that attack cut GPT-4o-mini on CommonsenseQA from 85.7% to 22.0%. This tests whether robustness learned on labeled distractors carries over to distractors built against the model's own weaknesses.