Corpus-Contrastive Token Weighting for Domain Pre-Training Against Entity Hallucination

When a general language model is further trained on biomedical text, every token counts equally in the loss. "The," "of," and "patients" get the same gradient weight as "tacrolimus," "eGFR," or "hazard ratio 0.82." Yet the tokens a model later gets wrong when it fabricates are mostly the rare, domain-specific ones: drug names, outcome measures, quantities, and which treatment arm a result belongs to. We propose a simple change. Before training, count how often each word appears across documents in a general corpus and in the domain corpus. Words that are much more document-specific in general text than in domain text, meaning words the domain uses often and the general web rarely uses, get a larger weight in the next-token loss. The weights are normalized so the total training signal stays the same. The cost is two document-frequency counts, with no auxiliary model.

The question is whether this nearly free statistic, a contrastive form of inverse document frequency (IDF, the classic retrieval weight that is large for words appearing in few documents), lowers closed-book factual errors on domain entities as much as more expensive model-based token weighting does, and whether it causes a predictable new failure. Our specific concern is that hedging words such as "may" and "suggests" are frequent and would lose weight, so the model might become more likely to overstate tentative evidence. The proposal is built to test both the benefit and that cost.

Core Claim

We claim that in continual pre-training of a decoder model on biomedical text, a fixed per-word weight based on corpus-contrastive IDF will reduce entity and attribute fabrications at matched token and compute budgets. We expect it to match a context-dependent weighting computed from the base model's own surprise, at a small fraction of the cost. We also predict that the variant which redistributes weight, taking it from common tokens to give to rare ones, will increase "certainty illusions," while a variant that only adds weight and never lowers it will not.

Let $\mathrm{df}_G(t)$ and $\mathrm{df}_D(t)$ be the fractions of documents containing word $t$ in a general corpus $G$ and a domain corpus $D$. We use the difference of log-IDFs:

$$\Delta(t) = \log\frac{1}{\mathrm{df}_G(t)} - \log\frac{1}{\mathrm{df}_D(t)} = \log\frac{\mathrm{df}_D(t)}{\mathrm{df}_G(t)}.$$

This is a log-likelihood ratio of document occurrence. It is a unigram, document-level version of "how much more surprising is this word to a general model than to a domain model." The training loss is

$$\mathcal{L} = -\frac{1}{\sum_i w_i}\sum_i w_i \log P(x_i \mid x_{<i}), \qquad w_i = 1 + \lambda\,\mathrm{clip}(\Delta(t_i), 0, c).$$

Here $x_i$ is the $i$-th subword. $t_i$ is the whole word that contains it, so every subword of a word shares that word's weight and the statistic does not depend on the tokenizer. $\lambda$ sets the strength, and $c$ caps outliers. The research file proposes the ratio $\mathrm{IDF}_G/\mathrm{IDF}_D$ instead. We use the difference because the ratio blows up for words like "patients" that appear in nearly every domain document, where $\mathrm{IDF}_D \to 0$.

Relation to Prior Work

Three groups of work motivate the proposal. Robust Hallucination Detection in LLMs via Adaptive Token Selection (HaMI) finds that hallucination signals in hidden states concentrate on a few tokens per answer, including on BioASQ. Its probe only detects errors after generation; we use the same sparsity observation to change what the model learns. Evaluating large language models on medical evidence summarization sorts medical summarization errors into attribute errors, fabrications, and "certainty illusions," and we adopt those categories directly as outcome measures. The paper studies only prompting of commercial models. A scalable framework for evaluating multiple language models through cross-domain generation and hallucination detection uses TF-IDF overlap and named-entity matching to score hallucination after the fact. We move the term statistic from evaluation into training.

The method itself comes from Prioritizing Image-Related Tokens Enhances Vision-Language Pre-Training (PRIOR). PRIOR upweights caption tokens that a text-only reference model cannot predict and reports a 19% average relative improvement over uniform next-token training for one model family. We keep its principle that ungrounded filler tokens dilute the gradient. We replace its trained reference model with corpus counts and move from vision-language captions to text-only domain adaptation.

Two papers constrain the design. BioBART continually pre-trains BART on 41GB of PubMed abstracts with a uniform loss and the general tokenizer. It finds that objectives that work on general text can hurt domain performance; removing sentence permutation helped. This warns that our weighting could also backfire, which is the reason for the certainty-illusion check. Finding Inverse Document Frequency Information in BERT shows that BERT's input embeddings correlate with IDF at r = 0.970, and that an auxiliary objective to predict IDF gave no ranking gain. That result could seem to contradict our proposal. We read it differently. Their objective asked the model to represent a statistic it already holds, on a ranking task where lexical rarity is already exploited. We do not ask the model to predict IDF. We change which prediction errors drive the weight updates during generative training, where the open question is whether the model's knowledge of the rare words themselves is good enough to recall them without retrieval. If our weighting also does nothing, that would extend their negative result to generation, which is useful to know.

Research Program

We will continually pre-train one open decoder base model of roughly 1B to 8B parameters on a fixed budget of PubMed abstracts. Every arm sees the identical token stream in the same order with the same optimizer and learning-rate schedule. Document frequencies for $D$ come from the training abstracts and for $G$ from a general web sample of comparable document count. We will report three things about the weight table: its correlation with base-model per-token loss on held-out abstracts, which word classes get the most weight, and the weights given to a fixed list of hedging terms drawn from Cochrane conclusion sections. That last item makes the certainty-illusion prediction checkable before any training.

The program has three questions. The first is whether contrastive IDF beats uniform loss on closed-book recall of domain facts and on downstream factuality. The second is whether it matches context-dependent model-based weighting, which is the stronger but costlier signal. The third is whether redistributing weight away from common tokens harms how well the model's stated confidence matches the evidence, and whether the upweight-only form avoids that harm.

Baselines

The do-nothing reference is the base model with no continual pre-training, fine-tuned identically for each downstream task. It shows how much any domain training helps.

The main naive competitor is uniform continual pre-training on the same tokens, the BioBART recipe applied to a decoder. This is the comparison that matters most, because weighting costs essentially no extra compute and must justify itself against simply training on the data. For the same recipe in an encoder-decoder, BioBART reports a ROUGE-L of 53.15 on MeQSum versus 51.05 for general BART-Large. That gives a scale for what domain pre-training alone achieves. We find no result on Lacuna for uniform decoder continual pre-training under our budget, so we will measure it ourselves over three seeds. Beating it means an improvement on the closed-book entity test and on PubMedQA that exceeds the spread across seeds.

The strongest existing method is PRIOR-style weighting, adapted so the frozen general base model serves as the reference. Its per-token loss on the domain text, normalized to mean one and clipped like our weights, becomes the weight. This is stronger than PRIOR as published, because no separate reference model has to be trained. The only extra cost is one forward pass of the base model over the corpus. That pass is substantial but not prohibitive, and we will report it in GPU-hours. Contrastive IDF "beats" this baseline if it comes within seed noise on the entity tests at a small fraction of the extra compute. If base-model surprise is clearly better, the cheap statistic loses, and we will say so.

The entity-weighted baseline upweights only tokens inside spans tagged by an off-the-shelf biomedical named-entity tagger, using the same $\lambda$ and normalization. It is the hand-built version of our idea.

Three ablations isolate the ingredients. Plain domain IDF, which drops the general corpus, tests whether the contrastive part matters. A shuffled-weight control keeps the exact distribution of weights but permutes them across word types. It tests whether gains come from which words get weight rather than from weight variance alone. The redistribute-versus-upweight-only comparison isolates the certainty-illusion mechanism. All arms get the same small sweep over $\lambda$, chosen on a held-out validation split that is separate from every test set.

Why not just…?

Why not just use retrieval-augmented generation? Retrieval is complementary rather than a substitute, and we will evaluate every arm with and without it. The evidence that retrieval alone is not enough is partial. Rethinking Hallucinations: Correctness, Consistency, and Prompt Multiplicity finds that retrieval lowers total hallucinations but makes the remaining errors more random under prompt rephrasing. Tang et al. find that giving the model longer source context hurt factuality. Retrieval also does not help when the model fails to recognize or reproduce the retrieved entity. If retrieval on top of uniform training erases all gaps between arms, the honest conclusion is that weighting matters only closed-book, and we will report it that way.

Why not just weight tokens by the base model's own surprise? This is the real threat. In continual pre-training the reference model already exists, which removes PRIOR's main cost. Base-model surprise is also context-dependent, while IDF is fixed per word. Our bet is that most of the useful signal is lexical and type-level, which the correlation analysis tests before training. The base-model-surprise baseline above settles the question directly. If it wins by a clear margin, the contribution shrinks to "a cheap approximation that recovers X% of the gain," and we will present it in exactly those terms.

Why not just upweight named entities? A tagger captures drug and disease names. It misses outcome measures, units, numbers, and relational words like "reduced" versus "increased," all of which appear in attribute errors. Contrastive IDF covers these without a tagger, but it may also upweight domain boilerplate. The entity-weighted arm, scored separately on entity errors and on numeric and relational errors, will show which signal covers which error type.

Why not just intervene at decoding time? Token-Guard reports an average F1 of 51.0 versus 28.3 for Llama-3.1-8B-Instruct across HaluEval, FinanceBench, DROP, CovidQA, PubMedQA, and RAGTruth. By its own account, it can keep a model faithful to what it knows but cannot supply missing knowledge. We will apply it on top of uniform-trained and weighted-trained models. The two approaches stack if the gain from weighting survives with decoding-time checks added.

Proposed Experiments

The primary test is a closed-book cloze built from held-out PubMed abstracts. We remove the conclusion sentence's high-contrast spans (entities, quantities, direction words) and score exact and normalized match. Results are broken down by how often each span's word appears in training, and the bins are defined by frequency rather than by $\Delta$, so the test does not mirror our own weights.

Downstream, we fine-tune each arm identically and evaluate on PubMedQA PQA-L. The reference points there are 68.1% accuracy for BioBERT with bag-of-words supervision, a 55.2% majority baseline, and 78.0% human accuracy, all from PubMedQA. We also evaluate on BioASQ question answering, on MeQSum (BioBART-Large ROUGE-L 53.15), and on the HaluEval QA split, where ChatGPT scored 62.59% per HaluEval.

Factual errors in generated text are counted in two ways: the entity-matching check from the MultiLLM-Chatbot framework, and clinician-style annotation using the Tang et al. categories on the 53 Cochrane reviews. For prompt robustness, we use Med-HALT under several prompt templates. Rethinking Hallucinations reports about 60% ambiguity there for Llama2-13B-Chat. We report correctness and consistency separately, as that paper recommends. To check forgetting, we measure perplexity on held-out general text.

Three outcomes would decide the proposal. The first is cloze and PubMedQA gains over uniform training that exceed the spread across three seeds, with no corresponding gain from the shuffled-weight control. The second is the size of the gap to base-model-surprise weighting, reported against the extra compute each method uses. The third is the certainty-illusion rate on the Cochrane reviews under redistributing versus upweight-only weighting. A higher rate under redistribution alone would confirm that frequency-based weighting trades entity accuracy for honest hedging. That would be a concrete warning for anyone designing token-weighted objectives for medical models.