Models fine-tuned for law, medicine or finance usually report high scores on their home benchmarks. Many of those benchmarks are near their ceiling and may have leaked into training data, so the scores say little about what the adaptation actually changed. This note proposes measuring something those scores hide: whether a specialized model can still ignore irrelevant text when that text is written in its own domain's language.
The hypothesis is that heavy domain adaptation teaches a model that domain terms are usually the important part of a prompt. That lesson helps on clean exam questions. It may hurt when a legal question is padded with a plausible but irrelevant statute, or a clinical vignette is padded with an unrelated lab value. If so, part of the "specialization tax" (the general skills a model loses when tuned for one field) has gone unnoticed because it shows up as loss of robustness to distracting context rather than as lower clean accuracy. The test is simple in outline. We compare each specialized model against the general-purpose checkpoint it was trained from. Both face the same domain questions, with matched amounts of irrelevant context of three kinds, and we analyze separately the questions the models have plausibly memorized and freshly written ones.
Core Claim
The work rests on three pieces of prior evidence. First, semantically coherent distracting sentences, found by an automated tree search, cut the accuracy of strong general models by roughly 45% on MMLU, CommonsenseQA, OpenbookQA and TruthfulQA. For example, GPT-4o-mini falls from 85.7% to 22.0% on CommonsenseQA (Breaking Focus; Adaptive Distraction). That work tested only general models on general benchmarks.
Second, robustness training for retrieval-augmented models explicitly leaves open how they behave with irrelevant context in law or medicine (Making Retrieval-Augmented Language Models Robust to Irrelevant Context). Third, the domain benchmarks that do stress long noisy documents observe distraction only incidentally. UDA tests only general models (UDA). ClinicBench notes that documents over 2,000 words and few-shot examples degrade clinical models, but it never injects noise in a controlled way (Large Language Models Are Poor Clinical Decision-Makers).
Our claim is the following. When we hold clean-question correctness fixed, specialized models keep less of their accuracy than their general-purpose parents under in-domain irrelevant context. They keep about the same or more under off-topic context. This asymmetry grows with the strength of adaptation and the scarcity of general replay data.
A second claim concerns evaluation. A distraction-budget curve separates models that a saturated benchmark scores as tied. The curve therefore restores headroom without needing new test items, which matters most in fields where expert-written questions are expensive.
Research Program
The central quantity is retained accuracy. We compute it only on items a model answers correctly without distraction, so that a weaker model is not penalized twice. For model $m$, distractor type $d$ and distractor budget $b$, measured in inserted tokens or tree-search depth, define
$$R_m(d,b) = \Pr\big[\text{correct under } (d,b) \mid \text{correct when clean}\big].$$
The specialization effect is then $\Delta(d,b) = R_{\text{spec}}(d,b) - R_{\text{base}}(d,b)$, computed on items that both models of a pair answer correctly when clean. The prediction is that $\Delta$ is negative for in-domain distractors and near zero or positive for off-topic ones.
We build three families of distractors for each domain.
The first family is off-topic coherent text. It is generated with the error-guided tree search of Adaptive Distraction, with the search restricted to topics outside the domain.
The second family is in-domain irrelevant text. It uses the same search, but candidates must contain domain vocabulary and are filtered by a judge model and by spot-checks from domain annotators. The filter confirms that the inserted text does not change the correct answer. Typical examples are a real but inapplicable rule, an unrelated drug interaction, or a neighboring table row in a financial filing.
The third family is in-domain conflicting text. This is relevant-looking text that contradicts the answer. We report it separately because the file identifies conflicting context as unsolved and different in kind from irrelevant context.
The budget axis runs from single sentences, as in the prior tree-search work, up to document-length padding. The long end connects to the long-document failures reported by UDA and ClinicBench.
Contamination enters as a stratification of items. For each benchmark we keep the original published items, which are plausibly seen during training. We pair them with fresh isomorphs: expert-rewritten or verified-paraphrased versions with altered surface facts and the same reasoning. Where the data allow, we also use post-cutoff items such as ClinicBench's Emerging Drug Analysis, on which nearly half of 22 models scored below chance.
There are two competing predictions about how memorized items respond to distraction. Answers that are memorized may resist distraction, because a strong prior overrides the context. Alternatively, they may break easily, because they rely on surface-form matching that the distractor disrupts. Either result tells us how far a model's clean benchmark score reflects recall rather than reasoning.
Saturation is addressed by recomputing the spread-versus-standard-error statistic of When AI Benchmarks Plateau along the budget curve. The aim is to show that clusters of models tied on clean scores spread out once distraction is added.
Released specialized models confound many choices: corpus, scale, preference tuning and replay data. To isolate cause, we therefore also run a controlled adaptation arm on a 7–8B open model. We continue pretraining it on a legal corpus and on a clinical corpus. Across runs we vary the replay fraction, with SaulLM's mixture of 2% general and 5% math data as one setting (SaulLM-54B & SaulLM-141B). We also vary whether a small fraction of noisy-context training examples is mixed in. These examples follow the SA-RetRobust recipe of 1,000 gold-plus-noise examples. This arm tests whether the in-domain robustness gap is a direct product of adaptation and whether cheap data choices prevent it.
Baselines
The naive reference point is clean accuracy on the original benchmarks: LegalBench-Instruct, the closed-ended medical sets inside ClinicBench, and FinHybrid and TatHybrid with retrieval. This is what papers currently report. The proposal beats it if the specialized-versus-base ranking under in-domain distractors differs from the clean ranking for at least one domain pair.
For the legal pair, the reported reference is SaulLM-141B at 75.34% mean balanced accuracy on LegalBench-Instruct, against GPT-4 at 73.35% and Llama-3 at 70.20% (SaulLM-54B & SaulLM-141B). This two-point clean margin is exactly the kind of gap we expect to reverse or widen under distraction.
The strongest existing method for measuring distraction is the adaptive tree search with error-guided perturbation (Breaking Focus; Adaptive Distraction). That work reports roughly a 45% average drop on general benchmarks. It also reports a 52% drop where DyVal2 produced 15.2%. We run it unchanged, with the same search budget, judge model and query count, on every domain item and every model. Our in-domain family uses the identical budget and differs only in the domain-vocabulary constraint. This ablation isolates the key ingredient. If the original off-topic search already yields the same $\Delta$, then in-domain distractors add nothing and our mechanism is wrong. We count the proposal as beating this baseline if in-domain distractors produce a significantly more negative $\Delta$ than off-topic ones at matched budget, using a paired bootstrap over items.
For retrieval-heavy finance, the baseline is GPT-4 with retrieval on FinHybrid, which scores 45.9 against 0.4 when fed the whole document (UDA). For defenses, we compare against Distraction-aware Answer Selection, a training-free method. It tells the reader to return "unanswerable" when unsure and down-weights answers from documents unlikely to have produced the query. It reports an average 64% exact-match gain on open-domain QA (Improving Zero-shot Reader). We also compare against the DPO training on distracted-versus-clean pairs from the tree-search papers, which reports nearly 50% gains in smaller models. Each defense gets the same tuning budget whether applied to the base or the specialized model.
No reported result was found on Lacuna for any domain-specialized model under controlled contextual distraction, so the domain-level comparison has no prior number.
Why not just…?
Why not just run the existing tree-search attack on specialized models? This is the obvious version of the idea, and it is the main baseline. On its own it cannot separate "specialized models are easier to distract" from "specialized models are weaker or differently calibrated." It also cannot tell whether domain-flavored noise is the specific weakness. Matched base-model pairs, conditioning on items answered correctly when clean, and the off-topic versus in-domain ablation are what turn a robustness number into a claim about specialization. If the off-topic attack alone reproduces the gap, we will report that and drop the domain-vocabulary mechanism.
Why not just evaluate on fresh, post-cutoff domain questions to escape saturation and contamination? Fresh items fix leakage but not measurement width. ClinicBench's emerging-drug set shows they are informative, but they are scarce and expensive in law and medicine. They are also saturated as soon as they are published, which is the "privacy myth" finding of When AI Benchmarks Plateau. Fresh items also say nothing about robustness. We use them as one stratum, not as a substitute. The direct test is whether distraction curves on old items rank models the same way clean accuracy on fresh items does. If they do, distraction is a cheap substitute for new items. If they do not, it measures something different.
Why not just add general replay or noisy-context training during adaptation and move on? This is a real threat to the importance of the work, not to its correctness. The replay and noise-mixing settings in the controlled adaptation arm test it directly. If a 5–7% replay mixture or 1,000 noisy examples closes $\Delta$ at no cost to domain accuracy, the practical recommendation becomes "always do this." The diagnostic contribution reduces to showing that current released models skipped it. We will report that outcome as such.
Why not just use perplexity-based contamination detectors? The file notes that post-hoc detectors struggle to separate reasoning from paraphrased contamination. We therefore treat contamination through a design choice, original versus isomorphic items, rather than a detector score. We include a detector only as a covariate.
Proposed Experiments
The first experiment covers the released model pairs. These are SaulLM-54B and SaulLM-141B against their general-purpose source checkpoints, scored on LegalBench through the LegalBench-Instruct format (LegalBench). They also include Meditron and PMC-LLaMA against their source checkpoints on the closed-ended and Referral QA parts of ClinicBench (AI-in-Health/ClinicBench). Finally, we evaluate general frontier models with retrieval on FinHybrid and TatHybrid. Each model is evaluated at five budget levels, three distractor families and two contamination strata. We use roughly 500 items per domain, sized by bootstrap power to detect a 5-point $\Delta$.
The second experiment is the controlled adaptation arm described above. It uses three replay fractions crossed with the presence or absence of noisy-context mixing. Each run is checked on the same domain items and on a general reasoning suite, so that the classic tax is measured alongside the robustness tax. We report BBH (BigBench Hard), where Specializing Smaller Language Models towards Multi-Step Reasoning saw chain-of-thought ability collapse after math specialization.
The decisive check for the main claim is the sign and size of $\Delta$ for in-domain versus off-topic distractors at matched budget, consistent across at least two domains and present in the controlled arm. A null result, with $\Delta$ near zero for both families, would falsify the claim and would also be useful: it would mean specialization does not trade away contextual robustness. Two checks serve the evaluation claim. One is whether model clusters that are tied on clean scores, by the spread-versus-standard-error criterion, become separable at moderate budget. The other is whether the original-versus-fresh difference in $R$ predicts which models drop most on post-cutoff items.
References
- Breaking Focus: Contextual Distraction Curse in Large Language Models
- Adaptive Distraction: Probing LLM Contextual Robustness with Automated Tree Search
- Making Retrieval-Augmented Language Models Robust to Irrelevant Context
- UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-World Document Analysis
- Large Language Models Are Poor Clinical Decision-Makers: A Comprehensive Benchmark
- When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation
- SaulLM-54B & SaulLM-141B: Scaling Up Domain Adaptation for the Legal Domain
- Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering
- LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
- Specializing Smaller Language Models towards Multi-Step Reasoning