When a large language model is adapted to a language with almost no text, one of the few usable resources is a bilingual dictionary. A recent method builds the new language's vocabulary from the dictionary headwords. It then gives each new token a starting vector, an embedding, equal to the average of the embeddings of the English words in its definitions. The weak spot is that English definition words are often ambiguous. A word glossed as "bank; shore; edge" inherits part of the vector for English bank, and in a model trained mostly on English that vector leans toward finance. The new riverbank token therefore starts life in the wrong neighborhood. We propose to resolve the ambiguous English gloss word to the sense its sibling glosses agree on, and to build the new token's starting vector from that sense alone. All of this uses only English-side resources and the dictionary entry itself, with no target-language text.
The question matters because, in the low-resource regime, the initial vector is most of what the model knows about a word. There is often too little target text to correct a bad starting point later. The question also has a sharp, testable form. If initialization errors from polysemy are real, fixing them should help most when the adaptation corpus is smallest. It should help most on tasks that require telling word senses apart, and should barely matter for named-entity tagging. If the gain is uniform, or disappears under a simple robust average, the idea is wrong. The experiments are built to say which outcome holds.
Core Claim
Dictionaries to the Rescue adapts Llama 3.1 and Gemma 2 to languages such as Manchu, Uyghur and Khmer using only about 21,000 dictionary words. It trains a tokenizer on the headwords. It then repeatedly removes long subwords so that the tokenizer falls back to shorter pieces, and aligns each target subword to source definition words with fast_align. Each target subword is initialized as a weighted average of the mapped source embeddings. The authors state that this averaging ignores one-to-many sense relations.
Our claim is that this blending is a measurable, fixable source of error. Replacing each ambiguous gloss word with a sense-specific vector, chosen by agreement with the rest of the dictionary entry, should improve sense-sensitive downstream behavior at small adaptation budgets. A robust but sense-blind aggregator should not deliver the same improvement. The prediction is a three-way interaction: the gain should be large on polysemy-sensitive tasks at near-zero adaptation data, small on entity tasks, and shrinking as target text grows.
Research Program
The method keeps the full fallback-and-align pipeline of Dictionaries to the Rescue and changes only the vectors that get averaged.
For every English gloss word, we first estimate whether it is ambiguous. We use the number of WordNet senses, the resource whose synsets Colex2Lang also builds on. If the word has one sense, its ordinary input embedding is used as before. If it has several, we build one candidate vector per sense: the average input embedding of that sense's synonyms and closely related words, restricted to words that are themselves unambiguous. This keeps each sense vector inside the model's own embedding space without needing any target-language text.
For a given dictionary entry, we pick the candidate sense whose vector has the highest cosine similarity to the other glosses of the same headword. "Shore" and "edge" pull bank toward its riverbank sense. That sense vector replaces the blended embedding in the average. Where a second pivot dictionary exists, its glosses join the agreement vote, for example Russian for Uyghur, or Hindi for Punjabi and Marathi. This borrows the logic of multi-pivot triangulation described in the research file: an ambiguity in one pivot language is unlikely to repeat in another. DHGNet found English is often not the best bridge language, so the second pivot is also a fair test of whether non-English glosses help on their own.
When an entry's glosses split cleanly into two unrelated groups, the target word is probably polysemous itself. In that case we keep a blended vector across the groups, but each group still uses its own resolved sense. The model can then separate the meanings in context, as contextual encoders do for English polysemy.
Alongside the method, we run a diagnostic that the field lacks. We measure how far each new token's initial vector sits from the embedding of its correct English translation, split by whether the gloss words were ambiguous. We then measure how much of that gap remains after language-adaptive pretraining, meaning continued training on whatever target text exists, at budgets of 0, 1k, 10k and 30k sentences. The 30k ceiling matches the 0.01GB regime of Vocabulary Expansion for Low-resource Cross-lingual Transfer. This tells us whether polysemy damage at initialization is washed out by adaptation, and how quickly.
Towards Multi-Sense Cross-Lingual Alignment of Contextual Embeddings is the closest methodological relative. It motivates the proposal by showing that averaging over senses produces mismatched geometry across languages. It clusters each word's contextual vectors with online k-means and aligns those clusters across languages, but it needs a large monolingual corpus to form the clusters and was tested only on English paired with German, Japanese and Chinese. Our method moves all sense resolution to the English side, where resources are plentiful. It touches only the initialization of a pretrained model, so it can run when the target language has nothing but a dictionary.
Parallel Tokenizers assigns translation-equivalent words the same vocabulary slot as their English counterpart. It has the same collapse problem in a harder form, and it needs machine translation that our target languages lack, so we treat it as context rather than a competitor.
Baselines
The naive reference point is random initialization of the new embeddings, followed by the same adaptive pretraining. A second naive point is a structural initializer, REINIT-SCRIPT, from Embedding structure matters. That paper showed this script-based heuristic rivals semantic initializers on part-of-speech tagging and NER. If our gains appear on sense tasks but not over this baseline on NER, that supports the claim that semantic initialization matters specifically for lexical meaning.
The strongest existing method is Dictionaries to the Rescue itself, run with the authors' fallback pipeline, the same dictionaries, the same Llama 3.1 8B base model, identical adaptive-pretraining data and steps, and the same downstream fine-tuning budget. Its reported strength is a Micro F1 of 94.87 on Manchu NER, against 28.00 for FOCUS. We will reproduce it before comparing. We found no reported result on Lacuna for this method on any word-in-context benchmark, so on sense tasks the baseline numbers will be our own reproduction.
Beating it means two things. First, a statistically reliable improvement over seeds on the word-in-context benchmarks at the 0 and 1k-sentence budgets. Second, Manchu NER no lower than the reproduced value, within seed variance.
The ablation that isolates the key ingredient is a sense-blind robust aggregator. Instead of the weighted mean, the same mapped source vectors are combined with a coordinate-wise median or an ℓ1-style objective. This follows the intuition of Cross-Lingual Word Embedding Refinement by ℓ1 Norm Optimisation, which treats polysemous mismatches as sparse outliers.
A second ablation drops ambiguous gloss words entirely. A third uses an oracle: a few hundred entries where a human picks the correct sense. This sets a ceiling on what better sense resolution could deliver. All variants share every hyperparameter except the initialization.
Why not just…?
Why not just let adaptive pretraining fix the initialization? This is the most serious threat. If a few thousand target sentences erase the difference, the method only matters at the very bottom of the resource scale. The budget sweep is designed to answer this directly. We pre-commit to reporting the budget at which the gap closes. A result where the gap closes by 1k sentences would make this a narrow finding, and we would say so.
Why not just use a robust average instead of resolving senses? The ℓ1 refinement work shows that down-weighting outliers improves cross-lingual alignment, particularly on distant pairs. A median over gloss vectors might capture most of the benefit without WordNet. This is exactly the sense-blind ablation. A robust average cannot help when the ambiguous word is the only gloss, or when it dominates the alignment counts, so we will report results separately for single-gloss and multi-gloss entries. If the median matches our method on both, we will adopt the median and drop the sense machinery.
Why not just rely on multilingual contextual models to sort out senses in context? Strengthening the WiC found that mBERT and XLM-R fine-tuned on English word-in-context data score only 52–55% on Hindi, near chance. SenWiCh found sense resolution transfers unevenly across ten low-resource languages. Contextual attention does not resolve senses for free across languages, and a model whose new tokens start in the wrong place has even less to work with.
Why not just train sense clusters on target text, as in multi-sense alignment? At 0.01GB or less, many polysemous words occur too rarely to form reliable k-means clusters, and our target languages often lack such text altogether. We include a k-means variant at the 30k-sentence budget only, to show where it becomes viable.
Proposed Experiments
Target languages fall into two groups. The first is the extremely low-resource group from Dictionaries to the Rescue, Manchu, Uyghur and Khmer, evaluated on perplexity and Manchu NER. The second is the SenWiCh languages that have both bilingual dictionaries and word-in-context data: Telugu, Kannada, Urdu, Marathi, Swahili and Azerbaijani. We add Hindi as a controlled case, replacing the model's Hindi vocabulary with dictionary-built tokens so that its 12,000-pair word-in-context set, built on 60 nouns, can be used under simulated scarcity.
For the word-in-context tasks, the adapted Llama is given a small classification adapter. It is trained in two regimes: zero-shot from English word-in-context data, and with 10% of the Hindi training pairs (about 700). Reference points come from the Hindi benchmark paper: 52–55% zero-shot, about 75% with 10% of the data, and 90% for MuRIL under full fine-tuning.
The first decisive check is the interaction. We test whether sense-anchored initialization beats both the mean and the median specifically on word-in-context accuracy at budgets of 0 and 1k sentences. We also test whether that advantage is several times larger than any change on NER. A uniform gain across tasks would point to generic regularization rather than polysemy.
The second check is per-item. Within the Hindi and SenWiCh test sets, the improvement should concentrate on targets whose dictionary glosses contained ambiguous English words. Targets with only unambiguous glosses serve as an internal control, since all methods give them identical vectors.
The third check is intrinsic. Among new tokens with ambiguous glosses, we count how often the nearest English neighbor is the correct translation, scored with CSLS (a nearest-neighbor score that corrects for vectors that are near everything) as in VecMap. We also count how often these tokens become hubs that attract many unrelated neighbors.
The oracle comparison bounds the remaining headroom. If oracle senses help much more than automatic resolution, the bottleneck is sense selection rather than the premise.
References
- Dictionaries to the Rescue: Cross-Lingual Vocabulary Transfer for Low-Resource Languages Using Bilingual Dictionaries
- Colex2Lang: Language Embeddings from Semantic Typology
- Cross-lingual Transfer for Text Classification with Dictionary-based Heterogeneous Graph
- How Can We Effectively Expand the Vocabulary of LLMs with 0.01GB of Target Language Text?
- Towards Multi-Sense Cross-Lingual Alignment of Contextual Embeddings
- Parallel Tokenizers: Rethinking Vocabulary Design for Cross-Lingual Transfer
- Embedding structure matters: Comparing methods to adapt multilingual vocabularies to new languages
- Cross-Lingual Word Embedding Refinement by $\ell_1$ Norm Optimisation
- Strengthening the WiC: New Polysemy Dataset in Hindi and Lack of Cross Lingual Transfer
- SenWiCh: Sense-Annotation of Low-Resource Languages for WiC using Hybrid Methods
- A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings