Teaching Clinical Models to Expand Abbreviations Missing From Every Dictionary

Clinical notes are full of abbreviations, often 30% to 50% of tokens, and many are ambiguous: "pt" can mean patient, physical therapy, or prothrombin time. Today's best systems treat expansion as a choice from a fixed list of candidate meanings. Those lists come from sense inventories or medical ontologies such as UMLS (the Unified Medical Language System, a large curated knowledge graph of medical concepts). When the right meaning is missing from the list, these systems cannot recover it. On the ShARe/CLEF 2013 benchmark, the best hybrid system scored 11.20% on abbreviations whose meanings had no ontology entry. The best language-model approach still reached only 30.42% on abbreviations it had never seen.

We propose to train a generative clinical language model to write out the expansion of an abbreviation. Two signals guide it: the surrounding note, and the letters of the abbreviation itself, which decoding enforces as a hard constraint. The model never chooses from an inventory. The key training trick is a form of entity masking. During continued pre-training on clinical notes, we take arbitrary clinical phrases, not only ones with known abbreviations, and replace them with synthetic abbreviations made by realistic formation rules such as initials, truncation, and dropped vowels. The model then has to reconstruct the original phrase. Because the synthetic short forms are new and do not repeat across the corpus, the model cannot memorize "abbreviation maps to expansion." It must learn the general skill of reading context and letters together. That skill is what transfers to meanings no dictionary contains.

Core Claim

The bottleneck in clinical abbreviation expansion is the closed candidate list, not the scoring model. This holds for ontology-constrained systems and for "non-sense-based" language models alike. A model pre-trained to reconstruct masked clinical phrases from synthetic short forms will expand meanings that are absent from both the inventory and its training data at much higher accuracy than candidate-ranking systems. It will stay competitive on covered meanings. A knowledge graph should be used after generation, to link an expansion to a concept identifier when one exists. It should not act before generation as a gate on what can be said.

The claim draws on three groups of prior work. The first motivates it. Clinical Acronym/Abbreviation Normalization using a Hybrid Approach built a tiered system of SVMs and profile matching that mapped abbreviations to UMLS identifiers. It found 221 test instances with no UMLS identifier and collapsed to 11.20% on them. Graph-constrained Reasoning restricts LLM decoding to a prefix tree of valid knowledge-graph paths. Its authors concede that a fact missing from the graph is impossible to output, even when the model's weights know it. We keep the idea of constrained decoding but move the constraint from "what the graph contains" to "what the abbreviation's letters allow," and that constraint is always available.

The second group supplies the closest method and the bar to beat. Improved Clinical Abbreviation Expansion via Non-Sense-Based Approaches frames expansion as fill-in-the-blank with a permutation language model. That model scores candidate strings without ontology glosses and avoids "length leakage," where the number of mask tokens reveals the answer's length. It reports 77.97% micro-accuracy on ShARe/CLEF and 30.42% on unseen abbreviations, but it still requires a candidate list. Our generative formulation removes the list. It also avoids length leakage by construction, because the decoder chooses the length.

The third group supplies the masking ingredients we combine. MeDAL pre-trains by swapping known full terms for their known abbreviations in 14.3M PubMed abstracts and asking the model to recover them. It is limited to dictionary pairs and to formal literature. EntityBERT masks 40% of cTAKES-detected clinical entities during continued pre-training on MIMIC-III and improves cross-domain transfer, but uses 100-token windows and never tests abbreviations. Zero-coverage mention randomization (Hongyu Lin et al., described in the file) replaces names with unique random strings so that named-entity models must rely on context. Our synthetic-abbreviation masking adapts that randomization to short forms: the replacement is random as a lexical item but realistic in its letter structure.

Research Program

The model. We start from BioBART, a BART model further pre-trained on 41GB of PubMed with text infilling. Its authors found that sentence shuffling hurt biomedical performance, so we keep discourse order intact. We continue pre-training on MIMIC-III notes with windows of 512 tokens or more, so the model can use section headers and document-level topic. Both Training without training data and Spasic's discourse analysis show that document-level context shifts abbreviation meaning.

The masking objective. Mixed into ordinary span infilling is the synthetic-abbreviation objective. We select multiword clinical phrases from two sources, cTAKES entity spans and noun chunks. Using both reduces dependence on one upstream tagger, a limitation EntityBERT inherits. Each selected phrase is replaced by a short form drawn from a formation grammar fitted to real (abbreviation, expansion) pairs in the meta-inventory of A deep database of medical abbreviations and acronyms. The model must generate the original phrase.

A real-pair share mirrors MeDAL reverse substitution. That share is capped and stratified by sense, so the frequent sense of an abbreviation does not dominate. This is the "false prior" Moon et al. documented: "ivf" occurs as intravenous fluid 3,000 times and as in vitro fertilization 0 times. We hold out a fixed set of real abbreviation senses entirely from pre-training. That set becomes our controlled test of dictionary-free expansion.

Decoding. We build a character-level automaton from the abbreviation, for example "each letter must begin, or appear in order within, successive words of the output." At each decoding step, tokens that would violate it are masked. This mirrors the prefix-tree constraint in Graph-constrained Reasoning, but it is built from the surface form instead of a graph. Inventory candidates, when available, enter only as a soft bonus in beam scoring, never as a requirement.

Linking. After decoding, a standard UMLS string-matching linker attaches a concept identifier when it finds one. Otherwise the system returns the free-text expansion and flags it as unlinked. A strict knowledge-graph system fails silently in that case; ours degrades gracefully.

Baselines

The naive reference point is a majority-sense lookup from the meta-inventory, which needs no context. This is what Normalization of Abbreviations/Acronyms: THCIB showed to be brittle, and it scores zero by definition on held-out senses. A second simple reference is BioBART fine-tuned on the labeled expansion data, decoded with no constraint and no synthetic-abbreviation pre-training. It tells us how much any of our additions matter.

The strongest existing method is the length-agnostic permutation language model of Jin et al. It is the best reported result on ShARe/CLEF, 77.97% micro-accuracy, and on unseen abbreviations, 30.42% accuracy. On MSH it reports 96.83% macro-accuracy. For ShARe/CLEF the prior hybrid best is 71.9% strict accuracy with 11.20% on uncovered terms, from Wu et al. For CASI, Moon et al. report accuracy rising from about 62% to 76%. We found no reported result on Lacuna for UMN or VUH accuracy. To make the comparison fair, we will re-run Jin et al.'s model with its candidate list drawn from the same meta-inventory our system sees. We will match pre-training data (MIMIC-III plus the same PubMed share) and compute, and give it the same hyperparameter search budget. We count our method as beating it if it matches it within one point on covered senses and exceeds it by a clear margin on held-out and uncovered senses. We set that margin at ten absolute points on held-out senses, to stay above seed variance.

We also run a graph-constrained baseline in the style of GCR, decoding only inside a prefix tree of inventory expansions. This makes the cost of hard knowledge constraints directly measurable. Moon et al.'s UMLS-relative augmentation, applied on top of our backbone, is a further knowledge-driven competitor.

Three ablations isolate the key ingredient. The first replaces synthetic-abbreviation masking with MeDAL-style real-pair reverse substitution at the same masking rate. The second replaces it with EntityBERT-style plain entity masking. The third removes the letter constraint at decoding. If the first ablation matches the full system on held-out senses, the central claim is false.

Why not just…?

Why not just use the bigger meta-inventory? The meta-inventory reports 96% sense macro-coverage on MIMIC-III, so closing the list might make open generation unnecessary. This is a real threat. The evidence against it is partial. Coverage on one ICU corpus does not equal coverage at other hospitals, where Moon et al. report 15% to 40% accuracy drops. Jin et al. note that under half of UMN abbreviations have ontology sense identifiers. We will measure inventory coverage directly on the gold expansions of ShARe/CLEF uncovered cases, CASI, and UMN. If residual uncovered mass is tiny everywhere, we will report that the contribution reduces to robustness against new or local jargon, and narrow the claim accordingly.

Why not just prompt a large general LLM with the letters and the note? Context-Aware Abbreviation Expansion Using Large Language Models expands over 70% of aggressive first-letter abbreviations in conversation, so this could work. We include a zero-shot and few-shot prompted LLM baseline with the same letter-constraint check applied after generation. Our model is small enough to run inside a hospital, which matters for protected notes. But if the prompted LLM wins on held-out senses, our story becomes a cheaper approximation rather than a better method, and we will say so.

Why not just apply Jin et al.'s model with a larger candidate list? A larger list helps only to the extent it contains the answer. The held-out-sense protocol removes those senses from every list by design, so a ranking model cannot score them. If list growth on the natural benchmarks closes the gap, the coverage measurement above will show it.

Why not just do Moon et al.'s UMLS augmentation? It needs the rare sense, and its relatives, to exist in UMLS. Its authors also report that it lowers accuracy on the source hospital. Our ablations test whether synthetic masking gives the cross-site gain without that in-distribution cost.

Proposed Experiments

The decisive experiment is the held-out-sense protocol. We remove a stratified sample of real abbreviation senses, spanning frequent, rare, and multiword, from pre-training data, fine-tuning data, and every candidate list. We then evaluate expansion on clinical sentences containing them. Accuracy is exact match after normalization. We also count UMLS-linked equivalence, and use clinician adjudication for unlinked outputs. If synthetic-abbreviation masking does not beat real-pair reverse substitution here, the idea fails.

On natural benchmarks we report ShARe/CLEF micro-accuracy overall and on its uncovered subset, MSH macro-accuracy, and CASI and UMN accuracy. For the uncovered subset we will annotate gold expansion strings wherever the benchmark gives only missing identifiers. For cross-institution transfer, we train on MIMIC-III and test on CASI and UMN, following Moon et al. We also report two diagnostics: the fraction of correct outputs with no UMLS identifier, which is the gap a graph-gated system cannot cover, and a calibration check on whether a low beam score flags wrong free-text expansions.

A second check guards against the model learning only letter matching. We shuffle or remove the context while keeping the abbreviation. Accuracy on ambiguous abbreviations should fall toward the majority sense if the model truly uses context.