Don't Forget the Long Tail! A Comprehensive Analysis of Morphological Generalization in Bilingual Lexicon Induction

Paula Czarnowska, Sebastian Ruder, Edouard Grave, Ryan Cotterell, Ann A. Copestake

Abstract: Paula Czarnowska, Sebastian Ruder, Edouard Grave, Ryan Cotterell, Ann Copestake. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

2019.

Introduction

Bilingual Lexicon Induction (BLI) is the foundational task of automatically finding word translations between languages, typically achieved by aligning word embedding spaces. While state-of-the-art models perform impressively on common words and lemmas, they struggle significantly with the "long tail" of rare morphological inflections.

Language follows a Zipfian distribution where most word forms are rare. While a human translator can easily generalize from a known verb root to a rare inflection, current BLI models often treat these rare forms as entirely new entities. This paper argues that standard evaluation metrics, such as the MUSE dataset, mask these deficiencies because they are heavily skewed toward high-frequency words and suffer from "semantic leakage," where different forms of the same word appear in both training and testing sets.

Methodology

To address these issues, the authors propose a quadripartite analysis framework that independently controls for four variables: word form frequency, morphology (grammatical features), lexeme frequency, and the lexeme itself. This allows for a granular understanding of where models fail.

To improve performance on rare forms, they introduce a hard morphological constraint during the training of self-learning BLI models. During the iterative alignment process, the model is restricted to mapping a source word to a target word only if they share the same morphosyntactic category (e.g., both are feminine plural adjectives). This ensures that the learned mapping is linguistically grounded.

Datasets and Benchmark

The study introduces a major new resource: 40 morphologically complete dictionaries covering 10 languages across the Slavic (Polish, Czech, Russian, Slovak, Ukrainian) and Romance (French, Spanish, Italian, Portuguese, Catalan) families. Unlike standard datasets, these include the full inflectional paradigm for every word, enabling the testing of "open-vocabulary" BLI, including out-of-vocabulary (OOV) words.

Analysis of Morphological Generalization

A key insight of the research is the "Frequency Trap": performance is not merely a function of word frequency, but of the rarity of the grammatical category. Models often fail on rare categories (like specific imperative forms) even when the root word is common.

Accuracy breakdown by specific morphological tags for Polish-Czech and French-Spanish, comparing frequent vs. rare forms.

For closely related languages, subword information (like n-grams) serves as a strong signal for morphology. However, Romance languages present a unique challenge due to their large verb paradigms (often exceeding 60 forms), making the task of selecting the exact correct inflection significantly harder than in Slavic languages.

Key Results

The evaluation reveals a dramatic performance collapse as word frequency decreases. For example, in the Catalan–Portuguese pair, accuracy drops from 83% for the top 10k words to just 40% for words ranked between 200k and 300k.

Accuracy of BLI models across different frequency bins for Slavic and Romance language pairs, illustrating the performance collapse in the long tail.

Implementing the hard morphological constraint led to consistent improvements across all Romance pairs and most Slavic pairs. Even when training data was restricted to only tagged words, the models generalized better because the resulting alignments were more linguistically sound. However, performance remained significantly lower for unrelated language pairs (e.g., Polish–Spanish), indicating that morphological generalization is most effective when languages share structural similarities.

Conclusion and Future Work

While morphological constraints offer a path forward, several challenges remain. "Syncretism"—where different grammatical forms share the same spelling—continues to confuse models and degrade embedding quality. Furthermore, the proposed method relies on the availability of morphological tags (e.g., from UniMorph), which may not exist for low-resource languages. Future work must address the mapping of complex morphology between unrelated language families, such as transitioning from agglutinative to isolating structures.

Summary updated

BibTeX
@inproceedings{czarnowska2019don,
  author = {Paula Czarnowska and Sebastian Ruder and Edouard Grave and Ryan Cotterell and Ann A. Copestake},
  title = {{Don't Forget the Long Tail! A Comprehensive Analysis of Morphological Generalization in Bilingual Lexicon Induction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=minQ2_id4Az},
}
Figure