Few Good Contexts: Sense-Covering Retrieval for Non-English Definition Generation

Language models write good English dictionary definitions and poor ones in most other languages. When human raters scored Llama-2 definitions on a 1–5 scale, English averaged 4.83 but Portuguese and German fell to about 1.8–2.0. Training one model on several languages at once gave no gain over training a separate model per language, so the word-to-definition mapping does not seem to transfer the way translation does (Multilingual Definition Modeling). This is the old knowledge-acquisition bottleneck of lexicography: hand-built resources such as WordNet and FrameNet take decades to produce and barely exist outside English. Our proposal is to stop asking the model to recall a word's meanings from its weights. Instead, give it a handful of real sentences from target-language text that use the word, and have it write the sense inventory from that evidence. This is retrieval-augmented generation, meaning a language model conditioned on passages fetched from an external text collection, applied here to dictionary building rather than question answering.

The step that makes this work is choosing which sentences to show, and that choice is a data quality-versus-quantity question. A raw search for a headword returns sentences dominated by its most frequent sense, often near-duplicates of one another, so the model never sees the rare senses. We claim that a small budget of sentences chosen to be non-redundant, typical of their sense, and spread across all senses will beat both closed-book generation and retrieval of many more raw sentences. The selection uses the semantic de-duplication and prototype methods developed for pretraining data. The largest gains should come on low-frequency senses, which current definition models handle worst.

Core Claim

For a target-language headword, cluster its retrieved usage sentences in embedding space. Remove near-duplicates. From each cluster, keep the sentences closest to its center. Give the generator these $k$ sentences and ask for one definition per distinct sense. We predict three things. First, curated retrieval beats closed-book generation, and the gain is far larger in German, Portuguese, Spanish and French than in English. Second, at a fixed context budget, curated sentences beat raw top-$k$ retrieval, and a small curated set (around 8 sentences) matches or beats a raw set four times larger. Third, the advantage is concentrated in recall of rare senses. A secondary, riskier prediction concerns the "no cross-lingual synergy" finding. Once the input is evidence rather than a bare headword, the task becomes "summarize these usages into a definition". That skill may transfer across languages, so mixed-language training should start to help.

Relation to Prior Work

Three papers motivate the gap. Multilingual Definition Modeling documents both the non-English collapse and the absence of transfer, and it supplies our main benchmark: roughly 730,000 definitions from DRAE, Duden and other professional dictionaries. LM-Lexicon clusters definition data into semantic domains, trains one expert model per domain, and merges them into a mixture of experts. It reports +7% BLEU and human accuracy of 0.85 against Claude-3-Opus at 0.80, but it works closed-book and only in English. Our method differs by clustering usage evidence at inference time for each word, not training data by domain. Multi-sense Definition Modeling using Word Sense Decompositions splits a word's static embedding into latent "sense atoms" and triples recall on 500 polysemous English words (fBLEU 0.355 vs 0.241). We pursue the same multi-sense goal, but we get senses from clusters of real usage rather than from a factorized vector. That should help with the fine-grained, low-frequency senses that both of these papers name as unsolved.

Two papers supply the selection method. D4 embeds a corpus with a small model, removes semantic near-duplicates, and keeps documents closest to cluster centers; for 6.7B-parameter pretraining this gives about 20% compute savings. SHED shows that 1–10% subsets chosen by cluster-level utility can beat the full instruction set (35.92% vs 33.16% on WizardLM). Both select data for training model weights. We apply the same logic per query, to the evidence a generator reads, where redundancy directly displaces information from a fixed context window.

Two papers constrain or neighbor the design. Lexical Diversity-aware Relevance Assessment for RAG (DRAG) shows that irrelevant retrieved text derails generation, for example a singer misdescribed as a priest. It needs a token-level calibration step to suppress such distractors. This warns us that adding more context is not free and that rare-sense sentences may also act as distractors, which our experiments must measure. RAKG uses retrieval to build entity-relation knowledge graphs, reaching 95.91% accuracy on MINE. It shows retrieval can drive structured resource construction, but it covers only named entities and does not curate its sources. Older lexical bootstrapping, such as Cross-Lingual Bootstrapping of Semantic Lexicons and Automatic Construction of Sememe Knowledge Bases via Dictionaries, found that aggressive filtering is what raised raw extractions to usable precision: projection precision rose from 30% to 68%, and distilled sememes outperformed HowNet. We take that lesson, filter hard, into a neural retrieval setting that needs no parallel corpus or existing dictionary in the target language.

Research Program

The datastore for each language is monolingual web and encyclopedic text. We first run D4-style semantic de-duplication over the whole collection. We then apply a strict leakage filter that removes any passage overlapping the evaluation dictionaries' definition text, since online dictionaries are widely copied across the web.

For each headword, a lexical search retrieves up to a few hundred candidate sentences; dense re-ranking is used for inflected forms. We embed each candidate by the contextual representation of the target word, so that sentences group by sense rather than by topic. Agglomerative clustering, with a distance threshold rather than a fixed number of clusters, groups the candidates into tentative senses. The context budget $k$ is spread across clusters with a floor of one sentence per cluster, so rare senses are not starved. Within each cluster we keep the sentences nearest the center and drop near-duplicates.

The generator is an open instruction-tuned model, fine-tuned per language on (curated contexts, gold sense inventory) pairs from the dictionary training split. It produces a list of definitions in the target language. Clustering is used only to choose the evidence. The generator decides how many senses to output, so clustering errors do not hard-code the inventory.

We will analyze where gains come from by stratifying results by sense frequency rank within the dictionary entry, by part of speech, and by language. We will also measure language-compliance failures, meaning output that drifts into English, which Multilingual Definition Modeling reports for zero-shot models.

Baselines

The naive reference point is closed-book generation by the same backbone, fine-tuned per language on headword-to-definitions pairs. This is the monolingual setup that Multilingual Definition Modeling found no multilingual setup could beat. The only reported figure on Lacuna for this regime is human evaluation of about 1.8–2.0 out of 5 for Portuguese and German (Llama-2), against 4.83 in English. No reported automatic-metric result found on Lacuna for monolingual fine-tuned models on this dataset. We will rerun the baseline ourselves and use the same human protocol so the comparison is direct. Beating it means a statistically significant gain in matched-sense BERTScore and in human rating, on the same test headwords.

The strongest existing method is LM-Lexicon's clustered-expert recipe (LM-Lexicon), retrained per target language on the same dictionary split with the same backbone size. Its reported English human accuracy is 0.85. No reported non-English result found on Lacuna for LM-Lexicon. We match training compute by giving it the parameters and steps our generator uses, plus the fine-tuning compute our method spends on retrieval-conditioned inputs.

The ablations isolate the key ingredient, which is curation. Every variant uses the same generator and the same $k$, and varies only which sentences fill the context. The variants are random sentences containing the word, raw top-$k$ by retrieval score, top-$k$ after de-duplication only, maximal-marginal-relevance diversity re-ranking, and our cluster-and-prototype selection. We also run raw top-$4k$ to test the claim that quality beats quantity. All variants get the same tuning budget, a small grid on a development split of identical size. Curation "wins" only if it beats maximal-marginal-relevance, not merely raw top-$k$.

Why not just…?

Why not just prompt a frontier model in the target language? This is the most serious practical alternative, and the file's evidence against it is partial. LM-Lexicon beat Claude-3-Opus and GPT-4-Turbo in English, and open models collapse in Portuguese and German. Neither result tests the strongest current frontier models in non-English languages. We will run a frontier model zero-shot and few-shot as an additional baseline, both with and without our curated contexts. If curated contexts lift the frontier model too, the contribution stands as an evidence-selection method usable with any generator. If the frontier model is already near gold without contexts, the closed-book bottleneck is narrower than claimed, and we would say so.

Why not just retrieve more raw sentences and let a long-context model sort them out? DRAG's results show irrelevant context actively misleads generation. Raw retrieval for a polysemous word is dominated by its frequent sense, so adding more raw sentences mostly adds redundancy. The raw top-$4k$ ablation tests this directly. If it matches curated top-$k$ on rare-sense recall, curation does not matter and the claim fails.

Why not just use maximal-marginal-relevance diversity re-ranking? This is a real threat, because generic diversity might recover most of the sense coverage without any clustering. It is included as a baseline at equal budget. If it ties our method, the honest conclusion is that diversity of evidence is what matters, and cluster structure adds nothing.

Why not just translate English definitions through aligned word senses? Bilingual sense alignments are scarce outside major language pairs (Cross-Lingual Bootstrapping of Semantic Lexicons). The non-transfer finding also suggests definitions are language-specific. We include machine translation of English WordNet glosses, for headwords with an aligned English equivalent, as a reference point. We expect it to lose most on language-specific senses, which have no English counterpart to translate.

Proposed Experiments

The main benchmark is the Spanish, French, Portuguese and German dictionaries from Multilingual Definition Modeling, split by headword. The English control uses WordNet and Oxford data from 3D-EX, as in LM-Lexicon. Generated senses are matched to gold senses by optimal assignment over BERTScore. From that matching we report sense recall, sense precision, and matched-sense BLEU and BERTScore, plus the fBLEU of Multi-sense Definition Modeling for comparability. Native-speaker raters will score a sample on the 1–5 scale used in that work.

The first decisive check is the quality-quantity curve. Sweep $k$ from 2 to 64 for raw and curated selection, and sweep datastore size from 1% to 100%. The claim holds if curated selection reaches its plateau at a small $k$ that raw retrieval needs several times more sentences to match.

The second check is rare-sense recall, stratified by sense rank. The claim predicts the curated-minus-raw gap grows with rank.

The third check is the language gap. The curated-minus-closed-book gain should be several times larger for Portuguese and German than for English.

The fourth check tests the transfer hypothesis by training one retrieval-conditioned model on all four languages and comparing it with per-language models. A positive gain would directly qualify the no-synergy finding. A null result means evidence alone does not make the task language-general.

The fifth check is a leakage audit. We rerun the evaluation after removing any retrieved sentence that shares a sequence of eight or more tokens with gold definitions. The main gains must survive this audit to count.