Borrowing Neighbors' Embeddings: Initializing New Domain Tokens in Small Language Models

A small language model adapted to medicine or law still reads a word like "acetylcholinesterase" or a clinical abbreviation as a chain of meaningless sub-word fragments. Adding that word to the model's vocabulary as a single token makes the model faster and gives it one place to store what the word means. The new token, however, needs a starting vector. The standard choice averages the vectors of the fragments it replaces, which encodes spelling rather than meaning. The authors of the strongest current method for choosing which domain words to add say plainly that finding a good starting vector remains unsolved.

This note proposes to borrow that starting vector from the word's semantic neighbors. The idea comes from semantic keyword expansion, where researchers train cheap word embeddings (vectors learned from which words appear in similar contexts) on a domain corpus and grow a small seed list into a full lexicon by taking each seed's nearest neighbors. We turn that machinery around. For each new token, we find its nearest neighbors in a domain-trained embedding space among words the model already represents well, and we initialize the token from those words' existing vectors inside the model. Distributional neighbors are known to confuse opposites such as likely and unlikely. We therefore mix the neighbor estimate with the fragment average, because the fragments carry the morphology ("un-", "non-") that distinguishes opposites. The claim is that this initialization lets a 1B–3B generative model add a domain vocabulary with little data, keep or improve task accuracy, and forget less general skill than current practice.

Core Claim

Vocabulary expansion for domain models has two separate decisions: which tokens to add, and how to initialize them. VEGAD addresses the first. It scores candidate words by how much loss gradient they receive at the embedding and output layers, and it shows that adding words indiscriminately wrecks general reasoning. In its experiments, GSM8K accuracy falls from 22.10% to roughly 6–9% under frequency-based expansion and only to 15.20% under gradient selection. VEGAD initializes by sub-token averaging and reports that Wikipedia-based alternatives did not help consistently.

We claim that most of the remaining forgetting and slow convergence comes from poor initialization, and that a domain-embedding neighbor estimate fixes a large share of it. Concretely, for a new token $w$ with sub-word pieces $p_1,\dots,p_m$ and domain-embedding nearest neighbors $n_1,\dots,n_k$ that already exist as single tokens in the base model, we set

$$ e_w = \alpha \cdot \frac{1}{m}\sum_{i=1}^{m} E[p_i] + (1-\alpha)\cdot \frac{\sum_{j=1}^{k} s_j\, E[n_j]}{\sum_{j=1}^{k} s_j}, $$

where $E[\cdot]$ is the base model's existing vector for a token, $s_j$ is the cosine similarity between $w$ and $n_j$ in the domain word2vec space, and $\alpha\in[0,1]$ trades morphology against distributional meaning. The same rule initializes the output-layer row for $w$, since VEGAD's gradients show both interfaces matter.

The prediction is specific. The benefit should be largest when fine-tuning data is scarce and the new tokens are rare. That is the regime where VE-KD saw its student degrade beyond about 60,000 added tokens, because rare tokens never received enough updates to recover from a bad start.

How the Pieces Fit

Semantic keyword expansion is the mechanism here, not the application. Two findings from that literature supply the method and one supplies a constraint. Mitra et al. (2016) showed that embeddings trained on local, specialized corpora beat general ones for query expansion. Theil et al. (2018) trained word2vec on 2.3 billion words of 10-K filings and doubled the volatility-explaining coefficient of an uncertainty lexicon, from 0.016 to 0.034. Together these justify training the neighbor space on the target domain rather than borrowing a general one.

Theil et al. also documented that antonyms land next to each other, which is the constraint. It motivates the mixture weight $\alpha$ and a diagnostic on polarity pairs, rather than trusting neighbors alone.

The closest methods for initialization come from cross-lingual work. Vocabulary Expansion for Low-resource Cross-lingual Transfer showed that an alignment-based initialization between old and new tokenizers beats plain averaging, and that tuning only the bottom and top two layers suffices with about 30,000 sentences. That work targets new scripts, and its candidates come from target-language statistics. We target domain terms and use semantic neighbors rather than tokenizer alignment. We reuse its layer-freezing recipe as one fine-tuning configuration.

On the selection side, Ali et al. (2025) is the one existing paper joining embedding-based expansion with a fine-tuned classifier. It found that 36% of embedding-flagged words were false positives, and it used expanded terms only as binary input flags, never as tokens. We use the neighbor space only to initialize, and we let VEGAD's gradient criterion decide which candidates enter the vocabulary. This keeps expansion noise from reaching the tokenizer.

Domain fine-tuning is the application regime. Hou et al. (2026) showed that a fine-tuned Llama-3.2-1B reaches 69.20% exact match and 87.16% category match on real MIMIC-IV discharge summaries. They left the tokenizer untouched, so rare medical terms remain fragmented, and they observed errors where the model picks a code one level too broad or too specific in the hierarchy. That setup is our main testbed.

Research Program

We will build a pipeline with four stages. First, train skip-gram embeddings on each domain corpus. Second, generate candidate tokens with VEGAD's corpus trie and gradient scoring, keeping its top-$K$ selection unchanged. Third, initialize each selected token by the rule above, with $k$ neighbors restricted to words that are single tokens in the base model and have high corpus frequency. Fourth, fine-tune under a fixed budget.

The mixture weight $\alpha$ is chosen once per domain by perplexity on held-out domain sentences containing new tokens, measured before any fine-tuning. That zero-step perplexity is itself a primary measurement of initialization quality.

We will also build an antonym diagnostic set. It contains polarity pairs mined from the domain corpus, such as likely/unlikely and positive/negative findings and the prefixed forms of clinical terms. On this set we measure whether the initialized model's next-token predictions separate the pair members.

The models are Llama-3.2-1B and 3B, the scale Hou et al. and EDCO studied. This also tests the generative setting that VE-KD left open. We sweep fine-tuning data at four sizes, from about 1% of the available training set up to all of it, and vocabulary additions from 1,000 to 20,000 tokens.

Baselines

The naive reference point is plain domain fine-tuning with the original tokenizer, both full-parameter and LoRA (a method that trains small low-rank weight updates instead of all weights). It is run on identical data with the same number of optimizer steps and the same learning-rate search. On legal Article QA, VEGAD reports this standard fine-tuning baseline at 26.70 BLEU. On MIMIC-IV, Hou et al. report 69.20% exact match for the 1B model trained this way with their two-phase data. We will reproduce both figures before comparing anything against them.

The strongest existing method is VEGAD with its default sub-token-mean initialization. It is the only published method that both selects domain tokens carefully and reports forgetting. Its reported numbers are 28.58 BLEU on Article QA, against 28.04 for Jieba frequency expansion, and 15.20% on GSM8K against a base of 22.10%. We run VEGAD with exactly the same selected token set as our method. Initialization is then the only difference, and both arms receive the same tuning budget for learning rate and steps.

Beating VEGAD means two things together. Task scores must be equal or better at every data size, with a clear gain in the smallest two data regimes. General-skill retention on GSM8K must be meaningfully higher than VEGAD's, recovering a substantial part of the gap to 22.10%. Every such comparison is reported with confidence intervals over several seeds.

A second initialization baseline is the Align strategy from the cross-lingual vocabulary paper. It is run with the same selected tokens and adapted to domain terms by aligning old and new tokenizations of the domain corpus. There is no reported result on Lacuna for Align on domain-specific (medical or legal) benchmarks, so we will report it fresh.

Frequency-based expansion with SentencePiece is included as the collapse reference, since VEGAD's reported 6–9% GSM8K figures come from this setting.

Three ablations isolate the key ingredient. Neighbor-only initialization ($\alpha=0$) tests whether morphology is needed to prevent antonym confusion. Neighbors drawn from a general-domain embedding rather than a domain-trained one test Mitra et al.'s locality claim in this new use. Random initialization shows how much initialization matters at all.

Why not just…?

Why not just prompt a frontier model and skip specialization? On the tasks in question the evidence is against it. OffensEval 2023 found zero-shot Llama 2 at 0.715 Macro-F1 against 0.829 for fine-tuned systems. PsyGUARD found a LoRA-tuned 6B model at 91.99% beating zero-shot GPT-4. Hou et al. found un-fine-tuned models below 1% exact match on ICD-10 coding. We include one zero-shot frontier run on MIMIC-IV only as a reference point, not as a competitor.

Why not just fine-tune without touching the tokenizer? This is the real threat. With enough data, fragmented tokens may carry domain meaning well enough that accuracy matches, as Hou et al.'s 69.20% suggests. If that happens at full data, the proposal's value narrows to two claims. The first is efficiency: fewer tokens per document and faster generation, which the cross-lingual paper measured at 2.6x to 4.5x for Burmese. The second is that good initialization is what makes this efficiency available without an accuracy or forgetting cost. The data-size sweep settles whether any accuracy advantage exists and where. We will report tokens per document and generation throughput alongside accuracy, so a tie on accuracy is still interpretable.

Why not just use the expanded neighbors to augment training data instead of adding tokens? Hou et al. obtained large gains from synthetic clinical variations, generated by prompting Claude. Substituting embedding neighbors into training sentences is a cheap version of the same idea, and it might deliver the semantic benefit with no tokenizer change. We therefore add an arm that uses the same neighbor lists for substitution-based augmentation under the original tokenizer, with a matched step budget. If this arm matches our method on accuracy, we will say so, and the contribution again reduces to efficiency with preserved general skill.

Why not just run VEGAD and train longer so initialization washes out? Initialization should indeed matter less as training grows. That is why the headline comparison is a curve over data size and training steps, not a single point. If the curves converge, the useful finding is where they converge and how much general skill was lost on the way. VEGAD's own 15.20% GSM8K figure indicates that the loss along the way is not small.

Proposed Experiments

The main experiment runs on MIMIC-IV coding with Hou et al.'s two-phase data. We report exact match, category match, and a count of hierarchy-level errors, to test whether meaning-bearing initializations for clinical terms reduce too-broad and too-specific picks. The second domain is VEGAD's legal Article QA, scored by BLEU. The third is PubMedQA, where VEGAD reported only a qualitative success and so gives us no reference number. GSM8K measures forgetting in every domain.

The decisive checks are three. The first is zero-step held-out perplexity on sentences containing new tokens, which should favor neighbor-mixture initialization before any training. The second is the gap between our method and VEGAD at the smallest data size, which should be the largest gap in the sweep. The third is the antonym diagnostic: the $\alpha=0$ ablation should confuse polarity pairs, and the mixture should not.

If zero-step perplexity shows no advantage, the mechanism is wrong and the project stops there. If the advantage appears at zero steps but vanishes after fine-tuning at every data size, the result is a clean negative finding about when initialization matters for domain vocabulary. That finding would also answer the open question VEGAD posed.