Biomedical text is full of entity names that a model can learn to recognize by spelling alone, and full of entity types that sit very close together in meaning: a sign versus a symptom, a disease versus a pathologic function, a clinical drug versus a pharmacologic substance. This proposal builds a continued pre-training objective that teaches a biomedical language model to tell such neighbouring types apart from context alone. The supervision comes for free. Some domain strings map to exactly one fine-grained type in the UMLS ontology (the large curated biomedical vocabulary). For those mentions in raw PubMed text, the type label is known without a human annotator or an automatic tagger. During pre-training we hide these mentions entirely and ask the model to predict their fine-grained type from the surrounding sentence. A margin is set by how close the true type and each wrong type are in the ontology, so confusing two sibling types costs more than confusing two distant ones.
Each ingredient covers a known failure of another. Mining unambiguous ("monosemous") words for training data dates back decades, but classifiers trained only on such harvested examples memorize the harvested words themselves and generalize poorly. Hiding the mention removes that shortcut. Entity-masking pre-training already exists, but its one fine-grained biomedical attempt traded a doubling of recall for a collapse in precision. We argue that collapse comes from coarse, flat type targets that are themselves produced by a tagger, which is exactly what clean, fine-grained, distance-aware targets should repair.
Core Claim
The claim is that masked type prediction on type-unambiguous UMLS mentions, trained with an ontology-distance margin, yields a biomedical encoder whose fine-grained typing errors fall mainly on siblings at a lower rate than both standard entity-masked pre-training and fine-tuning-time hierarchical losses. It does so without the precision loss seen in earlier entity-masking work. The specific new element is the pairing of label certainty at the granularity being predicted with mention hiding and taxonomic margins, all inside self-supervised domain pre-training rather than supervised fine-tuning. The claim is falsifiable in a precise way. If labels produced by a tagger or by naive string matching, given the same masking and loss, match the monosemous labels, then monosemy contributes nothing and the idea reduces to "add a margin loss to entity masking."
Prior Work and Where This Sits
Three papers motivate the proposal by exposing the gap. GPLSI-IXA: Using Semantic Classes to Acquire Monosemous Training Examples from Domain Texts harvested monosemous words from domain documents and grouped them into WordNet semantic classes for SemEval-2010 domain word sense disambiguation. Mixed with SemCor this reached 0.513 precision/recall, but harvested data alone fell to 0.380 because the classifier fit the harvested lemmas rather than their contexts. We keep the harvesting and remove the lemma from view. EMBRE: Entity-aware Masking for Biomedical Relation Extraction masked whole entities in PubMed and predicted their coarse concept type (gene, chemical, disease) and identifier. On BioRED relation extraction it raised F1 from 0.2640 to 0.3245, yet precision on the overall task fell from about 0.40 to 0.23. We differ in three ways: our labels come from monosemy rather than automatic annotation, our types are 21 fine-grained UMLS types rather than three coarse ones, and our loss reflects ontology distance. The MultiCoNER 2 shared task (SemEval-2023 Task 2) found that corrupting the entity mention hurts fine-grained NER far more (an 11.1% drop) than corrupting context (4.3%). In other words, current systems lean on the mention string to separate sibling types, which is the dependence we try to train away.
Two papers supply the methods we reuse. EntityBERT supplies the masking recipe. It continued PubMedBERT pre-training on MIMIC-III with entity-centric masking (40% of entities plus 12% random tokens) and reached 0.651 THYME TLINK F1 versus 0.638, but it reconstructs tokens, treats all entities alike, and depends on the cTAKES clinical tagger. Boosting Few-Shot Learning With Adaptive Margin Loss supplies the idea of class margins that scale with semantic similarity, which it derived from GloVe word vectors in image classification. We derive the similarities from the UMLS type hierarchy instead.
Several papers constrain the design. Distance-aware contrastive losses for fine-grained text classification, namely Not All Negatives are Equal (LCL, 60.1% vs 58.3% accuracy on 32-class Empathetic Dialogues) and LANE, show that weighting confusable negatives helps. Both, however, need an auxiliary network to estimate class similarity during training, which is too costly at pre-training scale. We therefore use a fixed, precomputed ontology distance. Hierarchical Losses and New Resources for Fine-grained Entity Typing and Linking shows that hierarchy-aware objectives help fine-grained typing, and it introduced MedMentions. Its encoder is a supervised CNN that sees the mention, so it competes with our method at fine-tuning time rather than at pre-training time. Reconnaissance d'Entités Nommées … Définitions des Types d'Entités improves the confusable MultiCoNER II Symptom class by 28.6 F1 by feeding type definitions to the model, which is a strong alternative way of separating siblings. Don't Neglect the Obvious found that adding unambiguous-word contexts raised the share of distinct sense embeddings from 22% to 68% but moved WSD F1 only from 74.0 to 74.5. That result warns us that monosemous data may change representations more than end-task scores, so we measure both. Finally, PiCO offers a way to learn from candidate label sets. We use its idea only as a comparison arm for ambiguous strings and do not use its prototype machinery.
Research Program
The lexicon. We first build a type-monosemy lexicon from the UMLS Metathesaurus. A surface string counts as type-monosemous if every concept it can name falls into the same one of the 21 MedMentions ST21pv semantic types. This criterion is looser than "maps to one concept": a string with three concepts that are all diseases still gives a certain disease label. It therefore raises coverage, which is the main known weakness of monosemous harvesting. Biomedical monosemy does not imply monosemy in ordinary English, since strings such as "lead", "cold" and "culture" also have everyday senses. We therefore drop any string that occurs in a general English lexicon or appears with high relative frequency in general web text. We report coverage per type, because types lacking unambiguous names will get little signal, and that imbalance is a result in itself. Exact string matching over PubMed abstracts then yields millions of labelled mention occurrences, and no tagger is involved anywhere.
The objective. Continued pre-training starts from PubMedBERT on PubMed abstracts. In each sequence, selected type-monosemous mentions are replaced entirely by mask tokens, with mention randomization (replacing the name with a random string, following Lin et al.) as an alternative arm. A small head on the pooled masked-span representation then predicts the type. Let $s_t$ be the score for type $t$, $t^*$ the true type, and $d(t^*, t)$ the normalized path distance between the two types in the UMLS semantic-type hierarchy. The loss is
$$ \mathcal{L}_{\text{type}} = -\log \frac{e^{s_{t^*}}}{e^{s_{t^*}} + \sum_{t \neq t^*} e^{\,s_t + \lambda\,(1 - d(t^*, t))}}, $$
where $\lambda$ sets the size of the margin. Close siblings, with small $d$, must be beaten by a larger gap than distant types. Unlike LCL or LANE, nothing in this loss is learned or recomputed during training. Standard masked-token prediction continues on all other tokens so that the model keeps its language ability. The total loss is the sum of the two terms, and the masking rates start from EntityBERT's reported optimum.
What we analyze. Beyond end-task scores, we measure sibling confusion: of a model's typing errors, the fraction that land on a sibling type. We also apply the MultiCoNER-style corruption test, measuring how much accuracy drops when the mention string is corrupted versus when the context is corrupted. If the method works, the gap between those two drops should narrow, because the model relies more on context.
Baselines
The naive reference point is PubMedBERT fine-tuned directly on each downstream task with no further pre-training. A second reference point is PubMedBERT given the same continued pre-training token budget on the same abstracts with ordinary random-token masking only. This controls for the simple fact of seeing more in-domain text.
The strongest existing competitor for the pre-training question is entity-masked pre-training with type prediction in the style of EMBRE. We reimplement it on our corpus: entities and types come from an automatic biomedical tagger, there are three coarse types, and the loss is plain cross-entropy. The token budget, steps and hyperparameter search match our method. Its reported BioRED numbers are relation-extraction F1 0.3245 (overall F1 0.2254, recall 0.2173, precision about 0.23), against 0.2640 for PubMedBERT. To beat it, our method must exceed its BioRED relation-extraction F1 under our reimplementation while keeping precision at or above the PubMedBERT level (about 0.40 in their setup). Raising F1 at the cost of precision, as EMBRE did, would not count. We also run an EntityBERT-style arm in which tagger-found entities are masked and their tokens reconstructed, with no type head.
For fine-grained typing, the strongest supervised competitors operate at fine-tuning time. The first is the hierarchy-aware typing loss of Murty et al., applied to a PubMedBERT encoder fine-tuned on MedMentions ST21pv. The second is the type-definition encoder of Lovón-Melgarejo et al.. Each gets the same fine-tuning data and tuning budget as our model, and we also test each stacked on our pre-trained encoder. I found no reported English MedMentions ST21pv NER number for these methods on Lacuna. The closest figure is 52.70 F1 for a fine-tuned domain encoder on the Vietnamese side of the parallel corpus En-ViMedNER, so all English numbers will come from our own runs.
Three ablations isolate the key ingredients. The first keeps everything but replaces monosemous labels with tagger labels for the same number of mentions. The second keeps the mentions visible instead of masked, which reproduces the GPLSI-IXA lexical-overfitting regime. The third sets $\lambda = 0$, giving a flat loss with no taxonomic margin.
Why not just…?
Why not just use tagger-produced labels, which cover every mention rather than only unambiguous ones? This is the most serious threat. Biomedical taggers are good, and more labelled mentions may outweigh cleaner ones. The label-source ablation settles it at matched mention counts, and a second run gives the tagger arm its full larger count. We expect monosemy to win at matched counts on sibling-confusion rate, since tagger errors concentrate exactly on neighbouring types. If the tagger arm wins once given more data, we will report that the useful parts are masking plus margins and that monosemy is not worth its coverage cost.
Why not include ambiguous strings as well, treating their possible types as a candidate set? This would recover lost coverage. We test it directly with a third label arm that adds type-ambiguous strings trained by maximizing the summed probability of their candidate types, a simplified version of the partial-label view in PiCO. If this arm matches or beats the monosemous-only arm, it becomes the recommended configuration, and the claim shifts from "unambiguous labels" to "label certainty handled correctly."
Why not just fine-tune with a hierarchical or definition-based loss and leave pre-training alone? With plenty of labelled data this may be enough. Both Murty et al. and Lovón-Melgarejo et al. report gains on confusable types. Our expected advantage lies with few labels and on unseen mentions, because the pre-trained model has already learned contextual type cues from millions of free examples. We run learning curves from 1% to 100% of the MedMentions training set. If the fine-tuning-only methods match us at every point, the pre-training stage is not justified.
Why not just keep the mention visible, so the model can use useful word-part clues like the antibody suffix "-mab"? Hiding the mention does throw away this morphology, a limitation the entity-masking literature already notes. GPLSI-IXA's drop from 0.513 to 0.380 shows what happens when harvested strings stay visible. The visible-mention ablation measures the trade-off on our data, and a further arm masks only the word stem while keeping the final subword, to see whether morphology can be kept without reopening the memorization shortcut.
Proposed Experiments
The main evaluation is fine-grained biomedical NER on MedMentions ST21pv (21 UMLS types), reported as micro and macro F1 and sibling-confusion rate. Results are also broken down by whether each test mention's string occurred in the pre-training lexicon. Gains on unseen strings are the clearest sign of context-based learning. The second evaluation is BioRED, where we report precision, recall and F1 separately for relation extraction and for the full pipeline, to test the precision-collapse hypothesis directly. The third, a check on transfer to clinical notes, uses THYME TLINK and the cross-domain negation sets used by EntityBERT, where the bar is EntityBERT's reported 0.651 TLINK F1. The fourth replicates the method's logic on SemEval-2010 Task 17, using WordNet semantic classes in place of UMLS types, to compare against GPLSI-IXA's numbers of 0.380 for harvested data alone and 0.513 combined. The fifth is a robustness probe on the medical classes of MultiCoNER II (including Symptom) using the mention-versus-context corruption protocol of the shared task.
Two results would decide the question. First, on MedMentions mentions whose strings never appeared in pre-training, the full model must cut the sibling-confusion rate relative to both the tagger-label arm and the flat-loss arm, across three seeds with non-overlapping confidence intervals. Second, on BioRED, the full model must raise relation-extraction F1 above the EMBRE-style reimplementation without letting precision fall below the plain PubMedBERT baseline. If the first holds but the second fails, fine-grained margins improve typing but do not fix the precision loss of relation extraction, and we will say so. If neither holds against the tagger-label arm, the monosemy component should be dropped.
References
- GPLSI-IXA: Using Semantic Classes to Acquire Monosemous Training Examples from Domain Texts
- EMBRE: Entity-aware Masking for Biomedical Relation Extraction
- SemEval-2023 Task 2: Fine-grained Multilingual Named Entity Recognition (MultiCoNER 2)
- EntityBERT: Entity-centric Masking Strategy for Model Pretraining for the Clinical Domain
- Boosting Few-Shot Learning With Adaptive Margin Loss
- Not All Negatives are Equal: Label-Aware Contrastive Loss for Fine-grained Text Classification
- LANE: Label-Aware Noise Elimination for Fine-Grained Text Classification
- Hierarchical Losses and New Resources for Fine-grained Entity Typing and Linking
- Reconnaissance d'Entités Nommées fondée sur des Modèles de Langue Enrichis avec des Définitions des Types d'Entités
- Don't Neglect the Obvious: On the Role of Unambiguous Words in Word Sense Disambiguation
- PiCO: Contrastive Label Disambiguation for Partial Label Learning
- En-ViMedNER: An English-Vietnamese Parallel Biomedical Corpus with UMLS Semantic Type Annotations