Randomized Withholding: Measuring What Domain Pre-Training Leaks Into Its Own Benchmarks

Specialized language models, such as medical models built by continuing to train a general model on millions of biomedical papers, are almost always scored on benchmarks made from the same literature they were trained on. PubMedQA, for example, turns a paper's title into a yes/no/maybe question and hides the abstract's conclusion. A model that read the full paper during training may simply remember the conclusion. It does not have to reason from the evidence. Nobody currently knows how much of the reported gain from domain pre-training is this kind of answer recall. The usual tools for finding out fail in this setting. Post-cutoff tests confuse memory with topic drift. Rephrasing hides memory without removing it. Statistical detectors perform at chance on pre-trained models.

We propose to answer the question with a controlled experiment instead of forensics. When a team does domain continual pre-training (further training an existing model on a domain corpus), it controls that corpus. We restrict the corpus to papers published after the base model was released, so the base model cannot have seen them even if its own training data is secret. We then randomly assign those papers to be included or withheld, at controlled repetition counts, and build benchmark items from both groups. Because assignment is random within the same time window, the accuracy gap between items from included and withheld papers is an unbiased estimate of leakage, meaning what the model gains from having read the source. The estimate carries no temporal-drift confound. The same design gives, at no extra cost, ground-truth labels for testing contamination detectors, a check on whether rephrasing or translation launders leakage, and a measurement of what weight merging keeps.

Core Claim

Randomized withholding inside a single domain continual pre-training run can separate three things that current audits mix together: answer leakage from the domain corpus, temporal distribution shift, and genuine domain skill. We will test four hypotheses. First, single-exposure leakage on PubMedQA-style items is measurable and grows with repetition. Second, a temporal-split audit misestimates it. Third, LLM rephrasing and translation hide leakage from surface signals but do not remove the accuracy benefit. Fourth, perplexity- and distribution-based detectors that look sound on general text lose accuracy on low-entropy technical text.

Each outcome is informative. A leakage estimate near zero, with a tight confidence interval, would be a useful negative result for anyone who reports domain benchmarks.

Relation to Prior Work

The closest work is Test of Time. It studies temporal decay on 20,277 arXiv math and physics papers, LiveCodeBench and FreshQA, and uses influence functions on OLMo2-7B-Instruct. It found a −3.83 pp post-cutoff drop for GPT-4o-mini on verbatim cloze questions, but no decay on transformed questions. Models were actually +2.19 pp better on post-cutoff physics. The source paper was the top-influence training document 77.5% of the time for cloze items but only 17.5% for transformed items.

That paper motivates our work: it shows that temporal signals are fragile and confounded by drift. It also constrains us: it rules out rephrasing as a clean fix. It observes leakage but never manipulates it, and its influence tracing was limited to 40 papers by compute. We manipulate exposure directly, so the ground truth comes from the design itself and not from expensive tracing.

Three further papers supply motivation. Is Your LLM Outdated? tracks bits-per-character on time-stamped Wikipedia, BBC and arXiv text. Its "ChatGPT effect", a drop in bits-per-character on arXiv after March 2024, is exactly the kind of drift that would bias a temporal-split audit of recent biomedical text. LiveCodeBench and LiveBench keep harvesting fresh items to stay clean. They assume time alone separates seen from unseen, which is the assumption we test.

Two lines of work supply the method and the controlled-contamination tradition. Investigating Data Contamination for Pre-training Language Models and Does Data Contamination Make a Difference? pre-train GPT-2 from scratch on a 3.3B-token slice of The Pile with benchmark items injected up to 20 times. They find accuracy peaks around 10 repetitions and that n-gram filtering fails. They inject benchmark items into general web text. We instead inject the source documents that benchmarks are built from, into a domain corpus, at 7B scale.

Quantifying Memorization in Continual Pre-training shows that continual pre-training on a narrow industry corpus (Nikkei, 700M tokens) memorizes much more than on Japanese Wikipedia. Maximum verbatim memorization rose from 15 to 27 characters, and the best membership-inference AUC was 0.689, from ReCaLL and LOSS. That paper measures memorization of text. It does not measure whether memorization changes benchmark answers, which is our question.

The detector literature constrains us most. Does Data Contamination Detection Work (Well) for LLMs? reports AUC near 0.50 for pre-training detection on Pythia and OLMo-2. It explains this with a "goodness-of-fit trap": a fluent domain model has low perplexity on all domain text. The same paper reports AUC 99.4 for BioMistral-NLU, but only after supervised fine-tuning. The continual pre-training phase remains unaudited, and that is where our testbed sits.

Two more methods are candidates for audit. Generalization or Memorization introduces CDD, which flags contamination when repeated samples collapse to near-identical outputs, and TED, which discards those peaked samples. CDD assumes clean models vary their outputs, an assumption that technical vocabulary may violate. Investigating Data Contamination in Modern Benchmarks introduces TS-Guessing, which probes whether a model can fill in arbitrary masked parts of an item. We evaluate all of these against real ground truth.

Finally, BioMistral supplies the pipeline we replicate: Mistral 7B Instruct v0.1, continually pre-trained on 3B tokens of PubMed Central and merged back with SLERP, TIES and DARE. It reports a 5.11% average zero-shot gain from SLERP merging on static benchmarks such as PubMedQA and MedQA, whose sources overlap its training corpus. It also translates its benchmarks into seven languages with GPT-3.5 Turbo. Our work asks how much of such gains is leakage, and whether translation hides it.

Research Program

Corpus. We start from PubMed Central open-access papers. The pool is those published after the base model's release plus a safety margin, roughly January 2024 to mid-2025 for Mistral 7B v0.1. We do not know Mistral's data, but it cannot contain papers published after its release. As a replication base with fully open data, we also use OLMo-2 7B, following Test of Time and Does Data Contamination Detection Work (Well) for LLMs?. For OLMo-2 we can additionally verify that pre-release papers are absent or present by direct search of its data.

The bulk of the roughly 3B-token corpus matches BioMistral's recipe. Into it we mix a set of target papers with structured abstracts, randomly assigned to one of six strata, all trained in a single run:

  1. Included once.
  2. Included four times.
  3. Included sixteen times. Together with the two strata above, this gives a dose-response curve that can be compared with the repetition curve reported for GPT-2.
  4. Source removed, neighbors kept. The paper itself is withheld, but documents that cite or restate its conclusion, such as reviews, commentaries and PubMed duplicates, remain in the corpus. This measures what title-level decontamination misses.
  5. Fully withheld. The paper and its identifiable neighbors are both removed. This is the clean control.
  6. Temporal holdout. The final two months of the window are withheld entirely, mimicking a conventional cutoff.

Randomization is blocked by journal and subject heading, so all strata except the temporal holdout share one distribution.

Items. From each target paper we build several kinds of item:

  • A PubMedQA-style item, following the construction in PubMedQA: the question is the title, the context is the abstract minus the conclusion, and the label is yes/no/maybe. Labels come from an LLM judge, audited by clinicians on a subset.
  • A no-context variant of the same item. This separates recall from reading: if the model answers correctly without the abstract, it is remembering.
  • A verbatim cloze item.
  • An LLM-generated multi-step item, following Test of Time.
  • A translated item, following BioMistral's approach.

Estimand. For item type $k$ and stratum $s$, leakage is

$$L_{k,s} = \mathrm{acc}_k(s) - \mathrm{acc}_k(\text{withheld}),$$

where $\mathrm{acc}_k(\cdot)$ is accuracy over items of type $k$ from papers in that stratum. Confidence intervals come from bootstrapping over papers. Randomization makes $L$ an unbiased causal estimate.

The temporal-audit estimate is the analogous gap between the included stratum and the temporal-holdout stratum. The difference between the two estimates is the bias of temporal auditing.

Merging. We apply SLERP, TIES and DARE merges with the base weights, as BioMistral does, and re-estimate $L$. This asks whether merging dilutes leakage or keeps it while restoring general skill.

Detection. Every target paper and item now carries a true label: seen $n$ times, or unseen. Detectors are scored against these labels.

Baselines

The naive reference points are the majority-class baseline and the base model without domain training. On the original PubMedQA the majority guess is 55.2% (PubMedQA). The base model is Mistral 7B Instruct v0.1, evaluated zero-shot on the same items with the same prompts. If our domain model's gain over this base on withheld items is near zero while its gain on included items is large, the headline gain is mostly leakage.

Domain-model references. The first is the public BioMistral 7B checkpoint and its SLERP merge, run on our items with identical prompting. BioMistral reports outperforming MedAlpaca and MediTron-7B by 9.0% and 9.6% across its suite. For absolute context on PubMedQA, the reported multi-phase fine-tuned BioBERT reaches 68.1% accuracy against 78.0% for human experts (PubMedQA). Our own continual pre-training reproduces BioMistral's recipe (token count, learning-rate schedule, merge settings), so our included, withheld and merged comparisons are directly comparable to the published model.

Audit baseline. The strongest existing way to estimate leakage without training-data access is the temporal split, as in Test of Time and LiveCodeBench. We implement it on the same model through the temporal-holdout stratum. It "wins" if its estimate falls inside the confidence interval of the randomized estimate. It loses if the difference is significant under a paired bootstrap.

Detector baselines. We run LOSS, perplexity/zlib, Min-K%, Min-K%++ and ReCaLL. The strongest competitor on domain continual pre-training is ReCaLL/LOSS, at AUC up to 0.689 on Nikkei (Quantifying Memorization in Continual Pre-training). We also run CDD and TED from Generalization or Memorization, and TS-Guessing from Investigating Data Contamination in Modern Benchmarks. Every detector gets the same sampling budget: the same number of generations per item for CDD, and the same reference model where one is needed. Thresholds are tuned on a held-out split of strata. We report AUC and true-positive rate at 1% false-positive rate.

We do not propose a new detector. The claim is about measurement, so no detector has to be beaten. Success means the testbed discriminates between detectors: for example, showing that CDD's false-positive rate on withheld technical items is higher than its rate on general text.

Key ablation. The design's key ingredient is randomization within a fixed time window. The ablation that isolates it compares the randomized leakage estimate with the temporal-split estimate from the same run. A second ablation compares the source-removed-neighbors-kept stratum with the fully withheld stratum. This isolates leakage that survives exact-source decontamination.

Why not just…?

Why not just use a post-cutoff temporal split? Across time, items change in topic, difficulty and writing style. Test of Time saw post-cutoff physics accuracy go up by 2.19 pp, and FreshBench saw LLM-written prose change predictability after March 2024. Either way, a temporal gap mixes leakage with drift. This alternative is a real threat, though: over a short window of biomedical abstracts, drift may be small, and the cheap audit may be good enough. Our temporal-holdout stratum settles this directly. If its estimate matches the randomized one, we will say so and recommend temporal splits for this domain.

Why not just run membership inference or CDD on the domain model? On pre-trained models these detectors sit near AUC 0.50 because of the goodness-of-fit trap (Does Data Contamination Detection Work (Well) for LLMs?). The best continual pre-training result is 0.689. CDD has never been tested on text whose correct answers are standardized terms. More importantly, a detector flags memorized text; it does not tell you how many accuracy points the memory buys. Our design measures both quantities and scores the detectors on the same papers.

Why not just n-gram-decontaminate the domain corpus against the benchmark? For PubMedQA-style items, removing papers whose titles match benchmark questions is cheap and would catch exact sources. But answers also live in reviews and restatements. The GPT-2 studies already found that n-gram filters leave influential material behind (Investigating Data Contamination for Pre-training Language Models). The source-removed-neighbors-kept stratum measures exactly the residual leakage this fix would leave. If that residual is zero, n-gram filtering is sufficient, and that is a publishable, practical answer.

Why not just rephrase or translate existing static benchmarks? Rephrasing drops influence-tracing hits from 77.5% to 17.5% without removing the underlying benefit (Test of Time). ChatGPT showed an estimated 41.5% leak ratio on HumanEval even with a rephrased version available (Generalization or Memorization). Our multi-step and translated item types test whether leakage persists under these transformations. Persistence is the laundering hypothesis.

Why not just use influence functions? They give per-document attribution, but Test of Time could afford only 40 papers. Randomization gives causal ground truth for thousands of papers at the cost of one training run.

Decisive Checks

There are five main runs: Mistral-based and OLMo-2-based continual pre-training, each with all six strata, followed by three merges each. We target about 1,500 target papers per stratum. The decisive checks follow directly from these runs.

The first is whether $L$ for included-once, no-context PubMedQA-style items has a bootstrap interval excluding zero, and whether it rises across one, four and sixteen repetitions. The second is whether leakage on the with-context variant is as large as on the no-context variant; if so, the benchmark rewards memory over reading. The third is whether leakage survives the multi-step and translated formats while cloze-style signals vanish. The fourth is whether the temporal-split estimate differs significantly from the randomized one. The fifth is whether SLERP merging reduces verbatim memorization, measured as the longest reproduced span as in the Nikkei study, while keeping leakage in accuracy. The sixth is how each detector's AUC and false-positive rate on the fully withheld stratum compare with its published numbers on general text.

We will release the corpora, stratum assignments and items, so other domain models trained on public PubMed Central can be audited with the same ground truth.