Abstract: Paula Czarnowska, Sebastian Ruder, Edouard Grave, Ryan Cotterell, Ann Copestake. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
2019.
Introduction
Bilingual Lexicon Induction (BLI) is the foundational task of automatically finding word translations between languages, typically achieved by aligning word embedding spaces. While state-of-the-art models perform impressively on common words and lemmas, they struggle significantly with the "long tail" of rare morphological inflections.
Language follows a Zipfian distribution where most word forms are rare. While a human translator can easily generalize from a known verb root to a rare inflection, current BLI models often treat these rare forms as entirely new entities. This paper argues that standard evaluation metrics, such as the MUSE dataset, mask these deficiencies because they are heavily skewed toward high-frequency words and suffer from "semantic leakage," where different forms of the same word appear in both training and testing sets.
Methodology
To address these issues, the authors propose a quadripartite analysis framework that independently controls for four variables: word form frequency, morphology (grammatical features), lexeme frequency, and the lexeme itself. This allows for a granular understanding of where models fail.
To improve performance on rare forms, they introduce a hard morphological constraint during the training of self-learning BLI models. During the iterative alignment process, the model is restricted to mapping a source word to a target word only if they share the same morphosyntactic category (e.g., both are feminine plural adjectives). This ensures that the learned mapping is linguistically grounded.
Datasets and Benchmark
The study introduces a major new resource: 40 morphologically complete dictionaries covering 10 languages across the Slavic (Polish, Czech, Russian, Slovak, Ukrainian) and Romance (French, Spanish, Italian, Portuguese, Catalan) families. Unlike standard datasets, these include the full inflectional paradigm for every word, enabling the testing of "open-vocabulary" BLI, including out-of-vocabulary (OOV) words.
Analysis of Morphological Generalization
A key insight of the research is the "Frequency Trap": performance is not merely a function of word frequency, but of the rarity of the grammatical category. Models often fail on rare categories (like specific imperative forms) even when the root word is common.
For closely related languages, subword information (like n-grams) serves as a strong signal for morphology. However, Romance languages present a unique challenge due to their large verb paradigms (often exceeding 60 forms), making the task of selecting the exact correct inflection significantly harder than in Slavic languages.
Key Results
The evaluation reveals a dramatic performance collapse as word frequency decreases. For example, in the Catalan–Portuguese pair, accuracy drops from 83% for the top 10k words to just 40% for words ranked between 200k and 300k.
Implementing the hard morphological constraint led to consistent improvements across all Romance pairs and most Slavic pairs. Even when training data was restricted to only tagged words, the models generalized better because the resulting alignments were more linguistically sound. However, performance remained significantly lower for unrelated language pairs (e.g., Polish–Spanish), indicating that morphological generalization is most effective when languages share structural similarities.
Conclusion and Future Work
While morphological constraints offer a path forward, several challenges remain. "Syncretism"—where different grammatical forms share the same spelling—continues to confuse models and degrade embedding quality. Furthermore, the proposed method relies on the availability of morphological tags (e.g., from UniMorph), which may not exist for low-resource languages. Future work must address the mapping of complex morphology between unrelated language families, such as transitioning from agglutinative to isolating structures.
Summary updated