Concept Deletion Pre-Training for Biomedical Language Models

Biomedical language models such as PubMedBERT learn by hiding random words in PubMed abstracts and predicting them back. This works well for frequent vocabulary, but it may teach the model little about how one medical concept implies another. When the model hides one mention of "warfarin", the same drug often appears unhidden in the title or the next sentence, so the model can simply copy it instead of reasoning from the surrounding clinical picture. Rare concepts, which are the ones that matter most in practice, are hidden too seldom to be learned well either way. Computer vision met the same problem in scene understanding and solved it by removing a whole object class from a scene and making the network recover it from what remains. We propose to bring that move into pre-training. Every mention of a chosen medical concept is deleted from a document, rare concepts are chosen as often as common ones, and the model must name the missing concept from its context. It names the concept rather than a particular spelling of it.

In two sentences: we change the masking step of biomedical pre-training so that it erases a whole concept across the document and asks for the concept back by identity, sampling concepts in a frequency-balanced way. We expect this to improve recognition and linking of rare concepts and to produce an internal map of which concepts depend on which. A cheap deletion test can then check that map against the curated relations in the main medical ontology, UMLS.

Core Claim

The claim has three parts, each testable separately. First, standard masked-word training in biomedicine is weakened by within-document leakage: a masked entity can be recovered from another unmasked mention of the same concept. We will measure how often this happens rather than assume it. Second, deleting all mentions of a concept at once forces the model to predict from co-occurring concepts. Choosing concepts with balanced frequency gives rare concepts enough training signal. Third, asking the model to recover the concept identity, scored against a fixed synonym-aware concept embedding, rather than one surface string, joins contextual pre-training with ontology-based synonym alignment. This addresses a limitation of each line of work. The predicted payoff is concentrated on low-frequency concepts and on entity linking in running text. We do not expect large gains on aggregate benchmarks, and the evaluation is designed around that prediction.

Research Program

The mechanism comes from vision. IDRNet groups pixels into semantic classes and zeroes out a whole class during training. It measures how other classes' losses respond, and it uses balanced deletion so that tail classes like traffic lights are deleted as often as sky. This raised an FCN baseline on ADE20K from 36.96% to 43.61% mIoU. Older TRECVID systems such as CLIPS-LSR-NII at TRECVID 2005 and NTU at TRECVID 2007 supply the underlying intuition: the confidence of one concept should be conditioned on related concepts. They did this with a separate classifier stacked on detector scores. We keep IDRNet's whole-group deletion and balanced sampling. We drop its learned relation matrix as a training component, because a transformer can represent concept dependencies internally. Instead, we reuse deletion as an analysis tool after training.

The domain side constrains the design. PubMedBERT showed that a domain vocabulary, pure in-domain text, and whole-word masking all help, and that general-domain text causes negative transfer. We therefore start from PubMedBERT and continue training on PubMed abstracts with its vocabulary. We change only the masking policy and the target. BioBART found that shuffling sentences hurts biomedical models because scientific discourse is ordered. Our deletion therefore never reorders text: it replaces spans in place and leaves the sentence sequence intact. SapBERT pulls UMLS synonyms together with metric learning. It works on isolated strings, which its own limitations note leaves it blind to sentence context. We use a frozen SapBERT as the concept dictionary for our identity target, so context comes from the document and synonym invariance comes from SapBERT.

The pipeline works as follows. We run a UMLS concept linker over the PubMed corpus to tag mentions with concept identifiers. The linker is a SapBERT nearest-neighbour linker over dictionary-matched candidates, and its tagging is noisy. We will report its estimated precision on a hand-checked sample. For each sequence, we sample concepts with probability proportional to $f_c^{-\alpha}$, where $f_c$ is the corpus frequency of concept $c$. Setting $\alpha=0$ recovers frequency-proportional choice, and $\alpha=1$ approaches uniform-over-concepts. All mentions of each sampled concept within the sequence are masked, up to the usual 15% token budget, with the remainder filled by standard whole-word masking. The model is trained on the usual token-reconstruction loss plus a concept-identity loss. The identity loss pools the hidden states over all masked mentions of $c$ into $h_c$ and scores it against frozen SapBERT concept embeddings $e_{c'}$:

$$\mathcal{L}_{\text{concept}} = -\log \frac{\exp(h_c^\top e_c/\tau)}{\sum_{c' \in \mathcal{N}} \exp(h_c^\top e_{c'}/\tau)}$$

Here $\tau$ is a temperature, and $\mathcal{N}$ contains the true concept plus hard negatives drawn from SapBERT nearest neighbours. Hard negatives follow SapBERT's own finding that hard-pair mining matters, with nearly 15% drops when it is removed. Because the target is the concept rather than a string, writing "pyrexia" where the text said "fever" is not penalized.

The analysis tool is a deletion probe. For a held-out document, we delete concept $A$ and measure the drop in the model's identity score for each other masked concept $B$. Averaged over the corpus, this yields a directed concept-dependency matrix. We then ask whether large entries coincide with UMLS associative relations such as "treats", "causes", and "may prevent" more often than with mere co-occurrence frequency. This tests whether the model learned clinical dependencies or just which words appear together, which is the correlation-versus-causation worry IDRNet raised for vision.

Baselines

The naive reference point is the PubMedBERT checkpoint itself, fine-tuned on each task with no further pre-training. This reproduces the published comparison point; PubMedBERT reports an overall BLURB score of 81.16.

The strongest and fairest competitor is PubMedBERT continued for exactly the same number of steps, on the same abstracts, with its native whole-word masking. The comparison uses identical batch size, learning rate schedule, and fine-tuning hyperparameter search budget, with five seeds per task. This controls for the trivial explanation that more training on PubMed helps. No reported result found on Lacuna for this continued-training variant, so we will produce it ourselves. Beating it means a statistically reliable gain, by paired bootstrap across seeds, on the rare-concept slices described below, with no loss beyond seed noise on the BLURB average.

For entity linking, the strongest existing method is SapBERT, which the file reports raising COMETA Acc@1 from 46.8% to 75.1%. We fine-tune it under the same protocol as our model. Our encoder is the base for a SapBERT-style linker fine-tuned identically, so any difference comes from pre-training.

For disease recognition, the context-fusion reference is the dictionary-guided graph-attention tagger of Lou et al.. It reports 89.95 F1 on BC5CDR-Disease against BioBERT's 86.56. We quote this number rather than rerun it. Our goal is not to beat a task-specific architecture with a plain encoder. Instead, we ask whether our encoder narrows the gap to it without a dictionary graph.

Three ablations isolate the key ingredients. The first is single-mention entity masking: one mention of a sampled concept is masked, others are left visible, and the target is the token reconstruction only. This is the stronger simple alternative. The second is all-mention deletion with frequency-proportional sampling ($\alpha=0$), which separates the effect of balancing. The third is all-mention balanced deletion with a token target only, which separates the effect of the concept-identity loss.

Why not just…?

Why not just mask whole entities, one mention at a time? This is the most serious threat, because it is a small extension of the whole-word masking PubMedBERT already found helpful, and it may capture most of the benefit. Our proposal rests on a specific mechanism, leakage from other unmasked mentions, so the first experiment measures leakage directly. We compute the fraction of single-mention entity masks whose concept appears unmasked elsewhere in the same sequence. We then compare single-mention against all-mention deletion at matched masking budget. If leakage is rare, or the two variants tie on the rare-concept slices, the central claim is wrong and we will say so.

Why not just upsample documents that contain rare concepts? Reweighting the data is cheaper than changing the objective and also gives tail concepts more exposure. It does not, however, stop the model from copying a visible mention, and it shifts the corpus distribution in ways that may hurt common-concept tasks. We will run it as an extra baseline at matched steps. Upsampling documents is the data-side analogue of our $\alpha$. If it matches our model on tail slices, balanced deletion adds nothing beyond exposure.

Why not just add context downstream, as a graph-attention or reranking layer at fine-tuning time? Lou et al. showed that a context-fusion layer over dictionary candidates gives large gains on BC5CDR-Disease. The stacking ensembles of Peng et al. won CHEMPROT with 0.6410 F1. These methods must be rebuilt for each task and need labeled data to learn the context weights. Concept knowledge placed in the encoder transfers to tasks with few labels. The test is the low-resource fine-tuning curve: if our gains vanish once enough labeled data is available, then downstream context modules are the better investment.

Why not just use a larger general-domain model? The file's evidence argues against mixing general text into domain training. PubMedBERT documents negative transfer from Wikipedia and BookCorpus, and Durrani et al. describe specialized patterns being drowned by generic data. We do not rerun this comparison. Our question concerns the objective at fixed domain data, which is orthogonal to scale.

Proposed Experiments

All models are BERT-base size and continue from PubMedBERT on the same PubMed abstracts. Each variant gets an equal step budget, and each is evaluated with five fine-tuning seeds.

The first experiment is the leakage measurement described above. It is a cheap check that should precede any training.

The main evaluation has two parts. The first is the full BLURB suite, used as a guard against regressions. The second is a set of frequency-stratified slices. We split test mentions in BC5CDR (chemical and disease), NCBI Disease, BC2GM, and MedMentions by the pre-training frequency of their linked concept into terciles, plus a set of concepts never seen in pre-training. We report F1 per slice for recognition and Acc@1 per slice for linking. The decisive prediction is a gain in the lowest tercile that the single-mention and upsampling baselines do not reproduce. We expect at most a small change in the top tercile.

For entity linking in running text, we fine-tune SapBERT-style linkers on MedMentions, where documents contain many concepts so context should matter most, and on COMETA and AskAPatient. Colloquial posts test whether context-informed concept prediction survives the register gap BioBART documented. No reported result found on Lacuna for MedMentions or AskAPatient numbers, so we report our own baselines there.

A low-resource curve fine-tunes on 1%, 10%, and 100% of the training data for BC5CDR-Disease and CHEMPROT. This tests whether the pre-trained context knowledge substitutes for labels.

Finally, the deletion probe computes the concept-dependency matrix on held-out abstracts. We measure how well it ranks UMLS-related concept pairs above unrelated pairs, using area under the ROC curve. We compare this with a matrix built from raw co-occurrence counts, and with the same probe applied to continued-MLM PubMedBERT. If our model's matrix tracks UMLS relations no better than co-occurrence counts do, the model has learned co-occurrence statistics rather than clinical dependencies, even if benchmark numbers improve. That finding would weaken the IDRNet-inspired interpretation.