Don't Forget the Long Tail! A Comprehensive Analysis of Morphological Generalization in Bilingual Lexicon Induction

Paula Czarnowska, Sebastian Ruder, Edouard Grave, Ryan Cotterell, Ann A. Copestake

Abstract

Human translators routinely have to translate rare inflections of words due to the Zipfian distribution of words in a language. When translating from Spanish, a good translator would have no problem identifying the proper translation of a statistically rare inflection such as habláramos, even if the lexeme itself, hablar, is common. In this work, we investigate whether state-of-the-art bilingual lexicon inducers are capable of learning this kind of generalization. We introduce 40 morphologically complete dictionaries in 10 languages and evaluate three state-of-the-art models on the task of translating less frequent morphological forms. We demonstrate that the performance of these models drops considerably when evaluated on infrequent morphological inflections and show that adding a simple morphological constraint at training time improves performance, proving that bilingual lexicon inducers benefit from better encoding of morphology.

Introduction

Bilingual Lexicon Induction (BLI) is the task of automatically finding word translations between two languages, typically by aligning their respective word embedding spaces. While current models claim high accuracy, these results are often deceptive. Most existing benchmarks, such as the MUSE dataset, are heavily biased toward high-frequency words. However, human language follows a Zipfian distribution: a few words are very common, but a vast "long tail" of words appears rarely. In morphologically rich languages, a single verb can have dozens of inflections. If a model can only translate the most common form, it fails to achieve true linguistic generalization.

Methodology

To move beyond simple frequency-based metrics, the researchers conducted a massive evaluation using a new experimental framework:

  1. Quadripartite Analysis: This framework independently controls for four variables: word form frequency, morphology (the specific grammatical tag), lexeme frequency, and the lexeme itself.
  2. Controlling for Lexeme: To isolate morphological knowledge from semantic knowledge, the authors created a task where the model must pick the correct translation from a set of candidates belonging to the same paradigm (e.g., different forms of the same verb).
  3. Morphological Constraints: The authors proposed a modification to the self-learning training process. At each iteration, the model is only allowed to align two words if they share the same morphosyntactic category (e.g., both are "3rd person plural future verbs").

Datasets

The study introduced a significant new resource: 40 morphologically complete dictionaries covering 10 languages (5 Slavic: Polish, Czech, Russian, Slovak, Ukrainian; and 5 Romance: French, Spanish, Italian, Portuguese, Catalan). Unlike previous datasets that might only include one or two forms of a verb, these dictionaries contain the full inflectional paradigm for every word. This allowed the authors to identify "semantic leakage" in standard benchmarks, where different inflections of the same word appear in both training and test sets, leading to inflated performance scores.

The Frequency Trap

A key insight of the paper is that BLI performance is not just about how common a word is, but how common its morphological category is. Models generalize well for frequent categories (like singular nouns) even for rare words, but they fail on rare categories (like imperative verbs) even for common words.

Figure 4: Accuracy of BLI models across frequency bins for Slavic and Romance language pairs, demonstrating the sharp performance drop in the long tail.

As illustrated in Figure 4, accuracy in the Catalan–Portuguese pair plummeted from 83% for the top 10k most frequent words to just 40% for words ranked between 200k and 300k. This trend is consistent across both Slavic and Romance language families.

Generalization Failure

In the "controlled lexeme" task, models achieved near 100% accuracy on frequent morphological slots but struggled significantly with infrequent ones. This suggests that models are often memorizing strings rather than learning underlying morphological rules.

Figure 5: Accuracy by morphological category for French-Spanish. Models perform significantly better on frequent categories (left) than on rare ones (right), even when the lexeme is controlled.

Key Results and Constraints

Simply enforcing that the model "only align words with the same grammar" during training acts as a powerful regularizer. Adding this morphological constraint yielded consistent performance improvements across all Romance language pairs. For example, French–Spanish accuracy rose from 47.8% to 50.2% on the standard task and saw even larger gains in the lexeme-controlled setup.

Conclusion

The performance of state-of-the-art BLI models is heavily dependent on frequency, failing to generalize to the morphological long tail. By introducing morphologically complete dictionaries and simple training constraints, this work demonstrates that better encoding of morphology is essential for robust bilingual induction. Future work will investigate using unsupervised morphological segmenters to provide these constraints for low-resource languages where gold-standard tags are unavailable.

Summary updated

Abstract

Human translators routinely have to translate rare inflections of words - due to the Zipfian distribution of words in a language. When translating from Spanish, a good translator would have no problem identifying the proper translation of a statistically rare inflection such as habl\'aramos. Note the lexeme itself, hablar, is relatively common. In this work, we investigate whether state-of-the-art bilingual lexicon inducers are capable of learning this kind of generalization. We introduce 40 morphologically complete dictionaries in 10 languages and evaluate three of the state-of-the-art models on the task of translation of less frequent morphological forms. We demonstrate that the performance of state-of-the-art models drops considerably when evaluated on infrequent morphological inflections and then show that adding a simple morphological constraint at training time improves the performance, proving that the bilingual lexicon inducers can benefit from better encoding of morphology.
BibTeX
@inproceedings{czarnowska2019don,
  author = {Paula Czarnowska and Sebastian Ruder and Edouard Grave and Ryan Cotterell and Ann A. Copestake},
  title = {{Don't Forget the Long Tail! A Comprehensive Analysis of Morphological Generalization in Bilingual Lexicon Induction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=minQ2_id4Az},
}
Figure