Is Less Really More? A Controlled Audit of Data Selection in Domain Continued Pre-Training

Teams that build a domain model for finance, medicine or code usually start from a general model and keep training it on domain text. This is called continued pre-training. A widely cited result says it is better to keep only a small, carefully chosen slice of that domain text: training on the "best" 10% beat training on all of it. This proposal tests whether that result is really about which data was chosen. It could instead come from three things that changed alongside the data. The first is training settings tuned for one condition and reused for the others. The second is the fact that a smaller subset means fewer training steps, so the model drifts less from its starting point. The third is that picking text "similar to the evaluation tasks" can pull near-copies of test items into training.

The idea is simple. Re-run the data-selection comparison under one shared training protocol. In that protocol, each data condition gets its own properly tuned learning rate and batch size, the number of tokens is controlled, and training text is checked for overlap with the benchmarks. We then measure how much of the "selection gain" survives. A domain-model builder needs to know whether their compute is better spent on data curation or on tuning, and today the literature cannot say.

Core Claim

The headline evidence comes from Efficient Continual Pre-training for Building Domain Specific Large Language Models. That paper continued training Pythia-6.9B on a 24-billion-token financial corpus and chose subsets two ways: by similarity to the target tasks (ETS-DACP) or by task-agnostic novelty and diversity (ETA-DACP). It reports a 12.45% improvement over the baseline with the task-similar 10% subset, against an 8.3% average improvement over base Pythia for the full-corpus FinPythia. Every condition shared one fixed training recipe.

Separately, Predictable Scaling Laws of Optimal Hyperparameters for LLM Continued Pre-training argues that a fixed recipe is the wrong choice for continued pre-training. It shows that the best learning rate and batch size depend on how much the checkpoint already "knows" about the target domain. It measures this as an equivalent pre-training compute, read off the checkpoint's starting loss, and fits scaling laws to predict good settings without full sweeps. That paper never varies data selection, and the financial paper never varies the optimizer settings. The research file flags this joint question as open (Gap 1).

Our claim is that a fair comparison must report the selection gain only after each data condition has had its own fair shot at good settings, and only after leakage is removed. We pre-register a falsifiable prediction: under that protocol, a random 10% subset recovers at least half of ETS-DACP's reported advantage over the full corpus. If it does not, curation is the real driver, and practitioners gain a justified reason to invest in it. Either outcome changes practice.

The method comes from the controlled-comparison tradition. On "Scientific Debt" in NLP gave older models the same backbone, corpus and fine-tuning as BERT, and the reported 8.8-point GLUE gap between BERT and ELMo shrank to 1.2 (78.0 vs. 76.8). Are GANs Created Equal? found that most loss variants performed alike once each received the same tuning budget. That equal-budget rule is the one we adopt. IsoPLM brought this kind of control to protein models but held the data fixed and varied the architecture; we do the reverse.

The Illusion of Generalization in Tabular Language Models supplies the third confound. Contamination and formatting explained most of a claimed domain gain there, and 8 of the top 10 datasets showed clear leakage. ETS-DACP selects text precisely because it resembles the task data, so it is the selection rule most exposed to the same failure.

Two results constrain the design. Personalized Benchmarking with the Ludwig Benchmarking Toolkit warns that one shared recipe can unfairly penalize some conditions. We therefore unify the tuning procedure and budget, not the settings themselves. The FineWeb Datasets shows that curation helps a great deal when training from scratch: educational filtering raised MMLU from 33% to 37% and ARC from 46% to 57%. That is a strong prior that selection is real, so we treat an outcome where selection survives as entirely plausible.

Research Program

We use the open Pythia checkpoints at 410M, 1.4B and 6.9B parameters, so the top scale matches FinPythia. Every run shares the same tokenizer, warmup-then-cosine schedule shape, sequence length, data order seed and evaluation harness.

The primary domain is finance. We rebuild a corpus following the composition described in the financial paper: about 83.5% financial news from Common Crawl and 16.5% SEC filings. Rebuilding is necessary because we cannot assume the original corpus is available, which also means the absolute numbers will differ from the published ones. A replication domain, biomedical text from PubMed abstracts as used by BioBART, tests whether conclusions transfer across domains.

There are five data conditions:

  • no continued pre-training;
  • the full corpus;
  • a uniformly random 10%;
  • ETS-DACP 10%;
  • ETA-DACP 10%.

Each runs under three training regimes:

  • the single fixed recipe, reproducing the original setup;
  • a per-condition grid search with an identical budget for every condition (six learning rates by two batch sizes);
  • per-condition settings predicted by the scaling-law method.

The grid is affordable at 410M and 1.4B. At 6.9B we rely on the predicted settings, after validating at the smaller scales that predicted and grid-optimal settings agree within seed noise.

To separate selection from step count, we add a token-matched arm: the full corpus truncated to the same number of tokens as the 10% subsets. We also add a compute-matched arm in which the 10% subsets are repeated for ten passes, so they see as many tokens as one pass over the full corpus. Before any training, all corpora are decontaminated by long n-gram overlap and embedding near-duplicate search against every evaluation set. We report results both before and after decontamination, so the size of the leakage effect is itself a finding.

The quantity we report is the selection gain:

$$\Delta_{sel} = S(\text{selected}, h^*_{sel}) - \max\big(S(\text{full}, h^*_{full}),\ S(\text{random}, h^*_{rand})\big)$$

Here $S(d, h)$ is the domain benchmark score after training on data $d$ with settings $h$, and $h^*_d$ is the best setting found for $d$ under the shared tuning budget. Selection is said to matter only if $\Delta_{sel}$ exceeds the spread across three seeds.

Baselines

The naive reference point is the base Pythia checkpoint with no continued pre-training. We evaluate it with the same prompts and fine-tuning as every other model. It anchors all gains, and if it ties the trained models on a task, that task is dropped from the selection analysis.

The strongest existing method is ETS-DACP as published. We run it in two forms. One reproduces its fixed recipe, to confirm we recover the qualitative ordering. The other is our tuned and decontaminated version. The reported figure to beat is the 12.45% improvement over baseline on financial tasks, which comes from Efficient Continual Pre-training for Building Domain Specific Large Language Models. The protocol "beats" the original analysis if it shows that a random 10% subset, or the full corpus with tuned settings, statistically ties this condition. The protocol "confirms" the original analysis if ETS-DACP stays ahead by more than seed noise after tuning and decontamination.

The full corpus with predicted per-condition settings is the strongest no-curation competitor. It follows Predictable Scaling Laws of Optimal Hyperparameters for LLM Continued Pre-training, whose authors report a peak benchmark average of 51.3 for their Dense-8B model on a different, general benchmark suite. That number is a sanity anchor only, not a direct comparison target.

The ablation that isolates the key ingredient is the random 10% subset. It shares everything with ETS-DACP, including token count and tuning budget, except the selection rule. The token-matched truncated-corpus arm separates "fewer steps" from "any subset at all."

For biomedical replication, BioBART-Large's reported MeQSum ROUGE-L of 53.15 (vs. 51.05 for BART-Large) indicates the size of domain gains to expect. No reported result found on Lacuna for DACP-style selection on biomedical text.

Why not just…?

Why not just run a full hyperparameter sweep for each condition and skip the scaling-law machinery? That is the gold standard, and we do it wherever it is affordable, at 410M and 1.4B. The scaling-law predictor is only a way to reach 6.9B, the scale at which the original claim was made. If predicted settings disagree with grid-optimal ones at small scale, we report that and limit our 6.9B conclusions to the fixed-recipe and predicted arms with that caveat. This alternative is not a competitor to the proposal; it is the reference the proposal checks itself against.

Why not just skip continued pre-training and fine-tune or prompt a general model? MediTOD found that a fully fine-tuned general Llama-3 8B reached the top in-domain NLU F1 (0.7139). The Illusion of Generalization in Tabular Language Models found a general instruction-tuned model recovered 92.2% of a specialized model's accuracy. This is a real threat to the whole enterprise, so the no-continued-training condition receives the same downstream fine-tuning as every other model. If it ties the trained models, the selection question becomes moot for that task, and we will say so.

Counter-evidence exists and keeps the question alive. A Comparative Study of PEFT Methods for Python Code Generation found that exposure to code during pre-training predicted success better than the choice of fine-tuning method.

Why not just accept FineWeb's evidence that curation dominates volume? FineWeb filters web text for models trained from scratch. Continued pre-training starts from a model that already knows a lot, which is exactly why its best settings shift. The two regimes may differ. If our audit finds selection survives, it extends FineWeb's conclusion to continued pre-training on solid ground. If it does not, it marks a boundary on that conclusion.

Why not just compare against a random subset under the original recipe? A random subset alone separates "selection" from "less data," but not from "settings that happened to suit small runs" or from leakage. Only the crossed design can show whether the published ordering flips once tuning is equalized.

Proposed Experiments and Decisive Checks

Domain evaluation for finance uses the FLARE financial suite, covering sentiment and named-entity tasks, as in the financial paper. For biomedicine it uses MeQSum ROUGE-L and BC5CDR entity linking from the BioBART suite. General retention, meaning how much broad ability the model keeps, is measured with MMLU and ARC.

The first decisive check is the interaction between selection rule and tuning regime. We fit a two-way analysis of variance in the style of One Model, Many Behaviors, which used variance decomposition to apportion out-of-distribution detection performance among training factors. If tuning regime explains more score variance than selection rule, the original claim was largely about optimization.

The second check is the change in $\Delta_{sel}$ before versus after decontamination. We expect this to be largest for ETS-DACP, because it selects by task similarity. The third check is whether the token-matched truncated corpus matches the random subset. If it does, "less is more" reduces to "fewer steps cause less forgetting," and the result should show up as better MMLU retention, not better FLARE scores.

The pre-registered success criterion, from the Core Claim, is that a tuned random 10% subset recovers at least half of ETS-DACP's fixed-recipe gain over the full corpus at 1.4B and 6.9B. Its failure is an equally publishable confirmation that curation is real.