Choosing the Words a Clinical Encoder Is Missing

Hospitals hold most of their patient information in free-text notes, and the small language models they can run on-site often read those notes badly. One cause is mundane. The model's vocabulary, its fixed list of word pieces, was built from web text. Clinical words like "fibrosis" get chopped into meaningless fragments, and ordinary words like "rejection" or "culture" carry a web meaning that differs from their clinical one. A striking recent result shows how much this matters. In exKidneyBERT, adding just six hand-picked words to a clinical BERT tokenizer before further training on 3,428 kidney pathology reports produced marked gains, while further training without those words gave minimal gains. The authors chose the six words by hand and flag automatic discovery of such terms as an open problem.

We propose to replace that hand selection with an automatic, budgeted procedure, and to test which part of it actually matters. The procedure draws on two kinds of evidence. The first is frequency contrast: terms that are far more common in a hospital's notes than in general English and that the existing tokenizer fragments badly become new vocabulary entries. The second targets words that are already in the vocabulary but whose clinical meaning has drifted from their general meaning. We detect these by clustering the substitutes a masked language model proposes for them, and we give them fresh embeddings anchored on their clinical neighbours. The question is whether such automatic selection, at a budget of a handful to a few thousand words, beats both the obvious frequency heuristic and simply training longer when the in-house corpus is only a few million words. We measure this on clinical extraction tasks where rare classes are the hard part.

Core Claim

In the small-corpus regime that real clinical departments face, the benefit of vocabulary extension depends on which words are added, not merely that some are added. We expect a selection rule combining three things to recover the exKidneyBERT effect automatically: domain specificity, tokenizer fragmentation, and a minimum occurrence count (new embeddings can only learn from the occurrences they see). We expect it to outperform a pure frequency-times-fragmentation rule at small budgets. We also claim that a second, previously untested channel exists. "Re-anchoring" means giving in-vocabulary words with drifted clinical senses a new embedding initialized from their clinical substitutes. We expect it to help beyond what contextual layers already recover, mainly on rare classes. Both claims are falsifiable by the controls below. If the simple heuristic matches our selector, we will report that as the finding.

Relation to Prior Work

The proposal reuses its methods from vocabulary discovery. Efficiently Mining High Quality Phrases from Texts scores phrases by how tightly their words co-occur and how sharply they separate from surrounding text. We use it to propose multi-word candidates such as "tubular atrophy" so that phrase boundaries are not left to chance. Large Scale Substitution-based Word Sense Induction induces word senses by clustering masked-model substitutes, and shows on PubMed that everyday words like "mosaic" acquire technical senses. That work stops at describing senses. We turn the sense clusters into a decision about which embeddings to reset and how to initialize them.

The architectural template comes from domain pre-training. exBERT adds an extension vocabulary plus small trainable modules beside a frozen BERT, but it takes the extension vocabulary as given. exKidneyBERT motivates the whole project. It never asks whether six other words, or six random frequent words, would have done as well. Without that control, we cannot tell whether the gain came from the terms or from giving the model fresh embedding rows to train. Two works bound the design. DoPAMine addresses data scarcity by mining more domain text from the web, an option unavailable for private notes, so we hold the corpus fixed. BioBART found that a standard pre-training objective, sentence shuffling, hurt biomedical performance. It warns that "more pre-training" is not automatically benign, which is why continued training without vocabulary change is a baseline here and not an assumption.

On the application side, extraction systems motivate the focus on rare classes and local models. MedSecId documents degraded performance on rare section types, and DRG-LLaMA does the same on rare diagnosis-related groups. Large Language Models Provide Human-Level Medical Text Snippet Labeling shows keyword pre-filters losing recall when clinicians use non-standard terms. Privacy rules push hospitals toward on-premise models, as in Medication information extraction using local large language models. That work adapts Llama 3.1 with LoRA, a method that trains small low-rank weight updates, and we treat it as the strongest larger-model competitor.

Research Program

The pipeline has three stages, run entirely on a hospital-sized corpus and a general English reference corpus.

The first stage proposes candidates. Phrase mining over the clinical corpus yields single- and multi-word strings. For each candidate string $w$ we compute

$$s(w) = \log\frac{p_D(w)}{p_G(w)} \cdot \big(f(w)-1\big) \cdot \mathbb{1}\left[c_D(w) \ge m\right],$$

where $p_D$ and $p_G$ are relative frequencies in the domain and general corpora, $f(w)$ is the number of pieces the base tokenizer splits $w$ into, $c_D(w)$ is its raw count in the domain corpus, and $m$ is a minimum count. The count floor exists because a new embedding row is updated only when its token appears, so with roughly 1.5 million words a rare term cannot be learned no matter how specific it is. We will sweep $m$ and report how the best value scales with corpus size.

The second stage finds drifted words. For every in-vocabulary word with enough clinical occurrences, we run substitution-based sense induction jointly over domain and general instances. A word's drift score is the fraction of its domain instances that fall in sense clusters with almost no general-corpus members. High-drift words, which we expect to include "rejection," "culture," and "positive," are re-anchored. We untie the word's embedding from its shared web-trained value and initialize it to the mean embedding of its top domain substitutes. No sense tagger is needed at inference, because within a single-domain deployment the domain sense dominates.

The third stage trains under a budget. The selected $K$ new strings receive embeddings initialized from the mean of their former pieces, and the model undergoes masked-language-model continued training on the same corpus. We test $K \in \{6, 50, 500, 5000\}$, with and without re-anchoring, using both plain embedding expansion and the exBERT-style frozen-base extension. The six-word budget is a deliberate replication point: we will check whether the selector, given the same budget, would have picked words like exKidneyBERT's.

The deliverable is the selector itself, together with a measured account of when vocabulary choice matters as a function of corpus size, budget, and class frequency.

Baselines

The naive reference is the base clinical encoder (Bio+Clinical BERT, the encoder used in CLEAR) fine-tuned directly on each task with no further pre-training. Above it sits continued pre-training with the original vocabulary, run for the same number of tokens and steps as every extension condition. exKidneyBERT found this gives minimal gains, so it sets the floor that any vocabulary method must clear.

The key ablation adds $K$ frequent but unselected domain words: high-count, non-fragmented words that are not domain-specific. It uses the same embedding initialization and training. This isolates whether gains come from the chosen terms or from simply adding trainable rows.

The strongest simple competitor ranks candidates by domain count times number of fragments. This is the rule a practitioner would write in an afternoon, and it is the most serious threat to the claim. It is run at every budget $K$ with the same phrase candidates and the same training.

The strongest existing method on the source task is exKidneyBERT's hand-chosen extension. The file gives no reported numerical result on Lacuna for exKidneyBERT, only "marked improvements," and its kidney corpus is private. We therefore replicate the recipe of hand-selecting the most diagnostic terms with a clinician on each public task. We give the human a fixed time budget and the same $K$, and compare.

A full domain-specific tokenizer retrained from scratch is included as the high-cost alternative. exBERT's frozen-base architecture is run with our vocabulary and with the frequency-heuristic vocabulary, so the architecture and the selection effects are separable.

As an outside reference for what a larger local model achieves, a LoRA-adapted Llama 3.1 8B follows the local medication-extraction setup. The file reports no numerical result for this setup on our tasks, so we measure it ourselves.

All encoder conditions share the same data, the same continued-training tokens, and the same downstream fine-tuning hyperparameter search of equal trial count, over five seeds. Beating a baseline means a macro-F1 improvement whose 95% bootstrap interval excludes zero on at least two of the four tasks, with no significant loss on the others.

Why not just…?

Why not just add the most frequent fragmented words? This is the real threat. It may capture most of the gain, because specificity and fragmentation are correlated in clinical text. Our bet is that it fails at small budgets, where it spends slots on frequent shorthand that context already handles while missing diagnostic terms of moderate frequency. It also cannot find drifted words at all, because those are not fragmented. The head-to-head at each $K$ settles this. If the heuristic matches the selector everywhere, the paper's conclusion becomes "use the heuristic plus re-anchoring," and re-anchoring is then tested on its own.

Why not just continue pre-training longer, or retrain the tokenizer? Longer training without new vocabulary is exactly what exKidneyBERT found unhelpful, and we test it at matched compute rather than assuming the result. Retraining the tokenizer discards the base model's learned embeddings for every changed piece, which is costly when the corpus is a few million words. The full-retrain baseline shows whether that cost is paid back.

Why not skip encoders and prompt or fine-tune a local LLM? Prompted pipelines like Self-Verification and mCODEGPT avoid training entirely, and LoRA-tuned Llama models run on one GPU. But their tokenizers fragment clinical words too, and many departments need models that are cheap to run over millions of notes. The LoRA Llama reference shows how much of the gap a small, well-vocabularied encoder closes. If the LLM dominates even on rare classes, we will say so.

Proposed Experiments and Decisive Checks

We use four public extraction tasks built on MIMIC data. The first is MedSecId section identification over 50 section types, scored by macro-F1 with rare sections reported separately. The second is radiology observation labeling following CheXbert's 14-observation setup. The third is diagnosis-related group prediction from the brief-hospital-course section following DRG-LLaMA, scored by macro-F1 and by accuracy on the rarest third of the 738 classes. The fourth is medication extraction from discharge summaries in the style of the cascade-of-classifiers task. To mimic the exKidneyBERT regime, continued-training corpora are built per note type at about 1.5 million words, then at 15 and 150 million to trace how the effect changes with scale.

The first decisive check is attribution. Gains from added terms should concentrate in test sentences that contain those terms. We split each test set by whether an added or re-anchored token occurs and report F1 on each part. Uniform gains would point to an extra-capacity effect rather than a vocabulary effect, which the frequent-word control should also expose.

The second check is the scale curve. We predict that the selector's advantage over the frequency heuristic shrinks as the corpus grows, since larger corpora let more candidates clear the count floor. Advantage that does not shrink would contradict the proposed mechanism.

The third check isolates re-anchoring. Re-anchoring alone, with no new strings, should improve rare-class F1 more than common-class F1. If it yields nothing beyond continued training, then the contextual layers already resolve drifted senses, and that channel is dropped.

The fourth check uses human judgment. A clinician blinded to the method rates the top-$K$ lists from each selector for diagnostic relevance. We then test whether these ratings predict downstream gain, which would tell practitioners whether expert review of an automatic list is worth its cost.