Does Curating Domain Text for Quality Make Models Easier to Distract?

When a general language model is adapted to a technical field such as finance or biomedicine, practitioners increasingly keep only the "best" domain text: the most complex, least redundant documents. This is a small fraction of the corpus, and it often beats training on everything. Real deployments, though, feed these models messy retrieved context: PDF layout debris, irrelevant paragraphs, near-miss documents. Models are known to collapse under exactly this kind of noise. This proposal starts from a question nobody has measured. Does quality-filtered continued pre-training on domain text make a model more vulnerable to distracting context, because it throws away the messy text that would have taught the model to ignore noise?

If the answer is yes, curation faces a real tension. The data that maximizes clean-benchmark accuracy is not the data that produces robustness. The fix we propose is a small online controller that chooses, at each stage of training, how much to draw from several data pools: complexity-filtered domain text, unfiltered domain text, domain text with inserted distractors, and general replay text. The controller plays a min-max game against a set of held-out probes (clean domain QA, distracted domain QA, general knowledge) and uses optimistic updates, which use a forecast of the next round's feedback, so that the mixture does not oscillate between objectives. In two sentences: first we test whether data curation trades clean accuracy for distraction robustness. Then we build and test a mixing controller that recovers both, and we check whether its "optimistic" update actually matters compared with simpler reweighting and a tuned fixed mixture.

Core Claim

The claim has two parts, and only the first is certain to be informative. The first part is diagnostic. Selection methods for domain continued pre-training improve clean domain QA, but they widen the gap between clean and distracted accuracy compared with full-corpus training. Data-Efficient Selection via Grammatical Complexity in Continual Pre-training of Domain-Specific LLMs (CDF-GC) keeps 20% of FinCorpus by grammatical complexity, beats full-data training by 1.7% on CFinBench, and never evaluates under distraction. Data-Efficient Selection via Grammatical Complexity in Continual Pre-training of Domain-Specific LLMs is therefore the method we stress-test. Two papers motivate the concern. Breaking Focus: Contextual Distraction Curse in Large Language Models shows that semantically coherent distractors found by tree search cut accuracy by about 45% on average, even though the models know the answer. UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-World Document Analysis shows that on financial documents, GPT-4 scores 45.9 with retrieval on FinHybrid but 0.4 when given the whole noisy document.

The second part is constructive. Mixture weights over a few pools, set online by optimistic no-regret dynamics against a worst-case probe, reach a better frontier between clean and distracted accuracy than any single curated set, and do so in one run rather than a grid search.

The optimistic ingredient is not decoration, and the reason is specific. Choosing a mixture to minimize the worst of several probe losses is a two-player zero-sum game. Standard (non-optimistic) multiplicative-weight dynamics in such games are known to cycle. Online Learning with Predictable Sequences and Faster Rates for Convex-Concave Games supply the method: both players use the previous round's feedback as a hint, which damps the cycling and speeds convergence. In our setting, cycling means that robustness data is over-sampled until clean accuracy dips, then under-sampled until robustness dips. Over a finite training budget, that wastes tokens.

Research Program

We first build the data pools for two domains. Finance uses FinCorpus, following CDF-GC. Biomedicine uses PubMed abstracts, as in BioBERT: a pre-trained biomedical language representation model for biomedical text mining and BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model. There are four pools. The first is a complexity-filtered pool, which is the CDF-GC selection. The second is the unfiltered remainder, which keeps the conversational and messy text that curation discards. The third is a distractor-injected pool, in which each domain document has semantically near but irrelevant spans from other domain documents spliced in, while the loss is kept on the tokens that depend on the original content. The fourth is a general-text replay pool that guards against forgetting.

The distractor-injected pool borrows its core idea from Focused Transformer: Contrastive Training for Context Scaling. That paper exposes attention to keys from unrelated documents during training, but it chooses the negatives at random. We use nearest-neighbor negatives instead, following the finding in Aligning Extraction and Generation for Robust Retrieval-Augmented Generation that distractors must be semantically competitive to teach robustness. We also keep the ordinary language-model loss rather than adding a memory layer.

The controller works as follows. Let $w_t$ be the mixture over the $K$ pools at round $t$ (a round is a few hundred optimizer steps) and $q_t$ a distribution over $J$ small probe sets. For each pool $k$, we estimate $u_{t,k}$, the per-token reduction in the $q_t$-weighted probe loss from training on pool $k$. The estimate is the inner product between a pool batch's update direction and the probe gradient, computed in the optimizer's update space as OPUS: Towards Efficient and Principled Data Selection in Large Language Model Pre-training in Every Iteration argues is necessary for AdamW. The mixer then plays optimistic Hedge,

$$w_{t,k} \propto \exp\Big(\eta\big(\textstyle\sum_{s<t} u_{s,k} + M_{t,k}\big)\Big), \qquad M_{t,k} = u_{t-1,k},$$

where $\eta$ is a step size and $M_t$ is the hint, meaning the last round's utility counted twice. The probe player runs the same update on the probe losses, but maximizes them.

Because a stale hint can hurt, as the file's summary of optimistic learning notes, we set $\eta$ adaptively from the running sum of squared hint errors $\sum_t \|u_t - M_t\|^2$. This is the quantity that governs optimistic regret, so the controller becomes more cautious on its own when utilities jump, for example right after a learning-rate warmup.

The controller differs from its two closest relatives on two points. Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methods (ADAPT) reweights samples reactively toward one fixed anchor set. OPUS selects batches reactively for one validation target. Neither has an adversarial probe player that balances several objectives, and neither has a look-ahead hint. Improving LLM General Preference Alignment via Optimistic Online Mirror Descent (ONPO) uses optimistic mirror descent for preference alignment over three to five offline iterations. We move the same kind of dynamics into continued pre-training, with pools as actions.

Baselines

The naive reference points are continued pre-training on the full domain corpus with random order, and the untouched base model. These fix the floor for clean accuracy, distracted accuracy and general retention.

The strongest static domain selector is CDF-GC, run on the same base models it used (Qwen2.5-0.5B and Llama-3.2-1B) at its 20% ratio and matched token count. Its reported result is a 1.7% gain over full-data training on CFinBench and 2.0% over perplexity and DSIR selection. To beat it we must match its clean CFinBench score within noise while shrinking the gap between clean and distracted accuracy. We have no reported number on Lacuna for CFinBench, MedQA or PubMedQA under inserted distractors, so this gap will be measured fresh for every method.

The strongest online selector is OPUS. It reports reaching peak continued pre-training performance with 0.5B tokens, against 3B tokens for random selection on SciencePedia. We run it with its default sketching and Boltzmann temperature, targeting the same clean domain probe, at equal tokens and equal wall-clock. ADAPT is run the same way, with our clean probe as its anchor set. It reports 6.1% better out-of-domain generalization than LESS.

A tuned fixed mixture over the same four pools is the baseline most likely to embarrass us. We give it a grid of about twenty mixtures, selected on the probes. Our single run must reach its best frontier point for the proposal to succeed.

Two ablations isolate the key ingredients. The first is the identical controller with the hint removed ($M_t = 0$, plain Hedge). This tests whether optimism reduces oscillation and wasted tokens. The second is the controller with the probe player frozen at uniform weights. This tests whether the worst-case game matters.

All methods share the same token budget, optimizer and learning-rate schedule. Each gets the same tuning budget of about ten runs for its own hyperparameters. Every continued pre-training checkpoint then receives the same light instruction-tuning stage before evaluation, so that differences reflect pre-training data and not post-training.

Why not just…?

Why not just prompt the model to ignore irrelevant context? Breaking Focus: Contextual Distraction Curse in Large Language Models reports that an "ignore irrelevant information" instruction was ineffective, while targeted training helped. Improving Zero-shot Reader by Reducing Distractions from Irrelevant Documents in Open-Domain Question Answering gets large exact-match gains (64% on average) only by adding abstention prompts and query-likelihood rescoring at inference. That method is complementary to ours, so we apply it on top of every checkpoint. If it erases all differences between checkpoints, pre-training data does not matter for distraction, and that negative result would be worth reporting.

Why not just fix robustness afterwards with Ext2Gen-style preference training? This is a serious alternative. Ext2Gen raises Llama3.1-8B-instruct's average score under noisy retrieval from 0.473 to 0.557. We will run Ext2Gen on each continued pre-training checkpoint. The proposal has value only if robustness learned during pre-training either survives Ext2Gen and adds to it, or reaches the same point with less post-training data. If Ext2Gen on the CDF-GC checkpoint matches our controller, we will say so and conclude that the robustness gap is cheap to repair after training.

Why not just tune a fixed mixture ratio? This may well work. The experiment answers it directly: we compare our one run against the best of about twenty grid runs. We also check whether the grid's best ratio transfers between the 0.5B and 1B models and between finance and biomedicine. If one fixed ratio matches the controller everywhere, the lasting contribution is the diagnostic finding and a recipe, and the controller is not needed.

Why not just use OPUS with momentum, which already looks at past updates? OPUS scores data against a single target in the optimizer's update space. The momentum it accounts for is inside the model's optimizer, not a forecast of how data utility will shift. OPUS has no mechanism for trading one objective off against another. Our comparison of OPUS pointed at a combined clean-plus-distracted probe with our controller tests exactly this.

Proposed Experiments

The first experiment is the diagnostic, and it decides whether the rest is worth doing. We run full-data training, random 20% selection, perplexity selection, CDF-GC, OPUS and ADAPT, all continued pre-training of Qwen2.5-0.5B and Llama-3.2-1B on FinCorpus and PubMed. For each checkpoint we measure clean accuracy and distracted accuracy. The finance evaluations are CFinBench and UDA FinHybrid and TatHybrid, using the benchmark's own noisy retrieval. The biomedical evaluations are MedQA-USMLE and PubMedQA; for reference, BioBERT reaches 68.1% on PubMedQA and humans 78.0%.

Distracted accuracy is measured two ways. One uses near-neighbor domain distractors. The other uses the tree-search distractors from Breaking Focus, generated once against the full-data checkpoint and frozen, so that no method is attacked adaptively and unfairly. MMLU measures forgetting. The prediction is falsifiable: if curated checkpoints show the same or a smaller clean-minus-distracted gap than full-data training, the premise fails. In that case we would report that curation does not cost robustness and stop.

The second experiment compares the controller against all baselines, the grid and the two ablations on the three-way frontier of clean accuracy, distracted accuracy and MMLU retention. The decisive check for the optimistic ingredient is to log the mixture weights over rounds. Plain Hedge should show measurable oscillation in the robustness pool's weight and in the probe losses. The optimistic version should show smaller oscillation and reach a given frontier point in fewer tokens. If the two trajectories and final scores are indistinguishable, we drop the optimism claim.

The last check tests the scientific-to-clinical gap described by BioBART. Evaluating the biomedical checkpoints on CovidDialog and MeQSum shows whether keeping the "messy" unfiltered pool also helps on informal clinical text. For reference, BioBART-Large reaches 53.15 ROUGE-L on MeQSum.