Does Domain Pretraining Entrench Claims That Later Turn Out Wrong?

Hospitals and labs increasingly build their own language models by taking a general model and continuing to train it on a large pile of in-domain text, such as pathology reports or COVID-19 papers. The reason is well supported: this step reliably beats general models on specialist tasks. Science and medicine also revise themselves, though. Early findings get contradicted, guidelines get replaced, and papers get retracted. This proposal asks whether the step that makes a domain model good also makes it hard to fix. The worry is that pouring concentrated, often repeated domain text into a model's weights hardens the claims in that text. When a claim later becomes obsolete, neither fresh evidence in the prompt, nor further training, nor deliberate forgetting would then remove it.

The work has two parts. First, we build a small, expert-verified, time-stamped dataset of biomedical claims that were later overturned, drawn from the literature that accumulated across the COVID-19 pandemic. Second, we run a controlled experiment. We vary how much domain text a model is trained on, then measure how hard each resulting model is to correct once the claim is overturned. We also test one cheap change to how the domain corpus is prepared, which is prefixing every document with its publication date. The hypothesis is that a model trained on dated text stores claims as tied to a time rather than as timeless facts, and so updates them more readily. If the entrenchment effect does not appear, that is also a useful result: it would mean practitioners can pretrain on domain text without paying a hidden cost in how easily the model can be corrected later.

Core Claim

We claim two things. First, the harder a claim resists correction, the more domain-concentrated the training was, and this can be measured as a dose-response curve. Second, attaching publication dates to documents at corpus-construction time flattens that curve, at no cost in domain accuracy.

Both claims are testable because we control the training corpus completely. We use an open model, OLMo, whose pretraining data (Dolma) is public, so we know what the model saw before our intervention. We also use CORD-19, a corpus of about 140,000 COVID-19 papers with publication dates.

Why This Gap and Not a Neighboring One

Two lines of work motivate the question.

The first comes from dataset construction. Continued training on specialist text works: Liu et al. continued masked-language-model training on the CORD-19 papers and raised scientific fact-checking precision on SCIFACT to 76.09%, against 46.6% for their baseline. The healthcare model-choice study further pretrained a clinical encoder on one million pathology reports and lifted its hardest task from 0.78 to 0.89 macro F1.

The second comes from work on obsolete knowledge. Paul et al. built SciUnlearn, a benchmark of 219 computer-science claims from papers in Dolma. Standard "unlearning" methods, which fine-tune a model to stop producing a given fact, only suppress the exact wording. After Gradient Difference, accuracy on paraphrased versions of the target claims barely moved, from 93.92% to 93.51%. They also found that more-cited papers resisted forgetting more.

Put side by side, these results raise a question neither answers. If repeated exposure hardens claims, then domain pretraining, which is repeated exposure by design, should harden them too. SciUnlearn observes citation-linked resistance only after the fact, on a fixed pretrained model, with a single correction route (unlearning), in computer science alone. It never varies exposure itself. Our proposal differs in four ways. We manipulate exposure directly. We work in biomedicine, which the SciUnlearn authors name as the missing domain. We treat correction as supersession by a newer claim rather than erasure. And we compare three correction routes on the same models.

Other work constrains the design. StateMem shows models anchoring on salient but superseded facts even when the current fact sits in the prompt, and reports that chain-of-thought can make this worse. We therefore cannot assume that supplying new evidence corrects the model, and we measure it directly. SituatedQA found retrieval models 15 points less accurate on updated facts than on stable ones, even with an updated corpus.

The date-prefix idea borrows from two sources. TimeLMs sliced Twitter into quarterly training corpora but never put dates into the model's input. Zhang and Choi showed that comparing evidence age against predicted fact lifespan cuts calibration error by 50–60%. Their method, however, works only on general-domain questions. We move the time signal into the corpus itself.

Research Program

The supersession dataset. We mine CORD-19 for claims about treatments, transmission, and risk factors that appear in early papers and are contradicted by later, stronger evidence. Examples include a small observational study followed by a randomized trial, or a preprint followed by a retraction or a failed replication.

Each item carries:

  • the original claim and its date;
  • the superseding claim, its date, and the evidence type;
  • a SCIFACT-style rationale sentence for each side (Wadden et al.).

Clinician annotators verify each pair. Alissa et al. show that expert-curated contradiction pairs cannot be replaced by generic augmentation in clinical trial text, which is why we do not rely on automatic mining alone. We target roughly 200 pairs, comparable in size to SciUnlearn.

Following SciUnlearn's design, each claim gets two question sets: one in the wording used during correction, and one set of held-out paraphrases. It also gets two sets of neighboring facts that should be preserved: facts from the same paper, and facts from papers it cites. We also count how often each old claim is repeated in the pre-cutoff corpus. This gives a second, natural axis of exposure, which is the within-corpus analogue of citation count.

The dose experiment. We fix a cutoff date $T$ that falls between the early claims and their refutations. Starting from OLMo-7B, we continue training on CORD-19 papers published before $T$ at four exposure levels:

  • zero domain tokens;
  • a 25% sample of the pre-$T$ papers;
  • the full pre-$T$ corpus;
  • the full corpus with the claim-bearing documents upsampled.

Every run is padded with general Dolma text to the same total token count, so compute is identical across conditions. Each level is trained in three corpus formats:

  • plain text;
  • text with the true publication date prefixed;
  • text with randomly shuffled dates prefixed.

The shuffled-date version tests whether any benefit comes from real time information rather than from extra formatting tokens.

The correction step. Each resulting model faces three correction routes after $T$, and we measure how much of the old claim survives each one.

The first is in-context override. The superseding abstract is placed in the prompt as perfect retrieval, and we check whether the model answers with the new claim.

The second is continued training on post-$T$ papers, in the style of the TimeLMs quarterly updates.

The third is unlearning with Gradient Difference, NPO, and SimNPO, each run with the retain sets described above.

The entrenchment measure. Our main measure is the fraction of paraphrase accuracy on the old claim that survives correction:

$$E = \frac{\text{Acc}_{\text{para}}^{\text{after}}(\text{old claim})}{\text{Acc}_{\text{para}}^{\text{before}}(\text{old claim})}$$

Here "para" means the held-out paraphrase questions, and "before" and "after" refer to the correction step. $E$ near 1 means the correction did not reach the underlying belief. $E$ near 0 means it did.

We report $E$ alongside three other quantities:

  • accuracy on the new claim;
  • accuracy on the neighboring facts that should be preserved;
  • SCIFACT verification F1, as a check on domain skill.

Baselines

Do nothing. The naive reference point is the zero-domain-token model, corrected by the same three routes. It shows how correctable the claims are when they come only from general pretraining. If domain training does not raise $E$ above this model's level, the entrenchment hypothesis fails. The general-pretraining starting point also serves as the "before" level for SciUnlearn-style comparisons. For the same model family, Paul et al. report paraphrase accuracy moving only from 93.92% to 93.51% under Gradient Difference. They also report MMLU falling from about 58 to about 39 when NPO is run without retain sets. We treat these as the expected failure pattern for unlearning, not a target to beat.

The strongest domain competitor. This is standard plain-text continued pretraining at full dose, the recipe Liu et al. used to reach 76.09% SCIFACT precision. It is the strongest competitor because practitioners actually use it and it maximizes domain accuracy. Our dated-corpus variant is trained on identical tokens, with identical steps, learning-rate schedule, and correction hyperparameters. The only difference is the date prefix. Tuning budgets for the correction step are matched: the same grid is searched for every model, and the search is scored on a separate validation split of claims.

Beating this baseline means two things together. The dated variant must show lower $E$ under at least two of the three correction routes. And its SCIFACT F1 must fall no more than one point below the plain-text model. For reference, Wadden et al. report 60.6 F1 in the perfect-retrieval setting and 39.5 F1 in the open setting for their pipeline. We will report our own numbers in both settings.

The retrieval-only competitor. This is the full-dose plain model given the superseding abstract in the prompt with no weight change. In RealTime QA, Kasai et al. report that retrieval lifts GPT-3 from 44.9% to 66.5% multiple-choice accuracy, but 34% of errors came from retrieval failures, in which the model fell back on stale knowledge. We therefore report the perfect-retrieval result and a result with retrieval deliberately removed for a random 34% of queries. The removal matches the reported failure share.

The shuffled-date ablation. This isolates the key ingredient. If shuffled dates help as much as true dates, the benefit comes from formatting, not time.

Why not just…?

Why not just use a frontier model with retrieval and skip domain pretraining? For some applications that is right, and the zero-dose arm measures exactly this option. The evidence in the file says it is not a general answer. The same healthcare study found a zero-shot 12B model at 0.54 macro F1 against 0.78–0.85 for fine-tuned domain encoders on a medium-difficulty task. StateMem shows that even with perfect retrieval, anchoring on superseded facts caused up to 63.5% of confirmed errors in memory benchmarks. Domain pretraining will keep being used, so its correctability cost needs measuring.

Why not just remove superseded documents before pretraining? This is the most serious alternative, and it fails for a structural reason. At training time, nobody knows which claims will be overturned later. Our cutoff design builds this constraint in: the models are trained before $T$ and corrected after. We also run an oracle-filtered arm, trained with hindsight removal of the later-refuted claims, as an upper bound. If dated training comes close to the oracle, curation-time dating is the practical substitute for knowledge nobody has.

Why not just prompt the model to reason about the timeline? StateMem reports that chain-of-thought sometimes lowered accuracy by rationalizing outdated facts. OAKS found that "thinking" modes raised susceptibility to distracting context. We still include a chain-of-thought variant of the in-context override route. If it closes the gap between the dated and plain models, the date prefix buys little that prompting cannot.

Why not just unlearn harder, with better retain sets? Paul et al. already used retain sets and still saw paraphrase accuracy almost unchanged. Their failure concerns reaching the concept, not protecting neighboring facts. Our dose curve tests whether this failure worsens with exposure, which stronger unlearning alone would not reveal.

Proposed Experiments and Decisive Checks

The first decisive check is the dose-response curve. We will plot $E$ against domain exposure for each correction route, and against within-corpus repetition counts for individual claims. A flat curve refutes the entrenchment hypothesis and supports domain pretraining without caveat. A rising curve confirms, in a controlled setting, the tension that SciUnlearn only observed correlationally.

The second check is whether true dates lower $E$ relative to both the plain and shuffled-date corpora while keeping SCIFACT F1. It only counts if the effect holds on the held-out paraphrase questions, where SciUnlearn showed surface-level methods fail.

The third check is transfer. We will test whether entrenchment measured on COVID claims predicts entrenchment on a second slice of claims from a different medical subfield in CORD-19, curated with the same protocol. This guards against results tied to one topic.

We will release all models, corpora, and the supersession dataset. Anyone could then repeat the test on other domains, such as the pathology or clinical trial settings in the file, where no reported result on correctability exists on Lacuna.