Abstract
Human translators routinely have to translate rare inflections of words due to the Zipfian distribution of words in a language. When translating from Spanish, a good translator would have no problem identifying the proper translation of a statistically rare inflection such as habláramos, even if the lexeme itself, hablar, is common. In this work, we investigate whether state-of-the-art bilingual lexicon inducers are capable of learning this kind of generalization. We introduce 40 morphologically complete dictionaries in 10 languages and evaluate three state-of-the-art models on the task of translating less frequent morphological forms. We demonstrate that the performance of these models drops considerably when evaluated on infrequent morphological inflections and show that adding a simple morphological constraint at training time improves performance, proving that bilingual lexicon inducers benefit from better encoding of morphology.
Introduction
Bilingual Lexicon Induction (BLI) is the task of automatically finding word translations between two languages, typically by aligning their respective word embedding spaces. While current models claim high accuracy, these results are often deceptive. Most existing benchmarks, such as the MUSE dataset, are heavily biased toward high-frequency words. However, human language follows a Zipfian distribution: a few words are very common, but a vast "long tail" of words appears rarely. In morphologically rich languages, a single verb can have dozens of inflections. If a model can only translate the most common form, it fails to achieve true linguistic generalization.
Methodology
To move beyond simple frequency-based metrics, the researchers conducted a massive evaluation using a new experimental framework:
- Quadripartite Analysis: This framework independently controls for four variables: word form frequency, morphology (the specific grammatical tag), lexeme frequency, and the lexeme itself.
- Controlling for Lexeme: To isolate morphological knowledge from semantic knowledge, the authors created a task where the model must pick the correct translation from a set of candidates belonging to the same paradigm (e.g., different forms of the same verb).
- Morphological Constraints: The authors proposed a modification to the self-learning training process. At each iteration, the model is only allowed to align two words if they share the same morphosyntactic category (e.g., both are "3rd person plural future verbs").
Datasets
The study introduced a significant new resource: 40 morphologically complete dictionaries covering 10 languages (5 Slavic: Polish, Czech, Russian, Slovak, Ukrainian; and 5 Romance: French, Spanish, Italian, Portuguese, Catalan). Unlike previous datasets that might only include one or two forms of a verb, these dictionaries contain the full inflectional paradigm for every word. This allowed the authors to identify "semantic leakage" in standard benchmarks, where different inflections of the same word appear in both training and test sets, leading to inflated performance scores.
The Frequency Trap
A key insight of the paper is that BLI performance is not just about how common a word is, but how common its morphological category is. Models generalize well for frequent categories (like singular nouns) even for rare words, but they fail on rare categories (like imperative verbs) even for common words.
As illustrated in Figure 4, accuracy in the Catalan–Portuguese pair plummeted from 83% for the top 10k most frequent words to just 40% for words ranked between 200k and 300k. This trend is consistent across both Slavic and Romance language families.
Generalization Failure
In the "controlled lexeme" task, models achieved near 100% accuracy on frequent morphological slots but struggled significantly with infrequent ones. This suggests that models are often memorizing strings rather than learning underlying morphological rules.
Key Results and Constraints
Simply enforcing that the model "only align words with the same grammar" during training acts as a powerful regularizer. Adding this morphological constraint yielded consistent performance improvements across all Romance language pairs. For example, French–Spanish accuracy rose from 47.8% to 50.2% on the standard task and saw even larger gains in the lexeme-controlled setup.
Conclusion
The performance of state-of-the-art BLI models is heavily dependent on frequency, failing to generalize to the morphological long tail. By introducing morphologically complete dictionaries and simple training constraints, this work demonstrates that better encoding of morphology is essential for robust bilingual induction. Future work will investigate using unsupervised morphological segmenters to provide these constraints for low-resource languages where gold-standard tags are unavailable.
Summary updated