Aligning Word Vectors on Low-Resource Languages with Wiktionary

Mike IzbickiCOLING 2022

TL;DR

This paper introduces a new, high-quality bilingual lexicon collection derived from Wiktionary, covering 298 languages, many of which are low-resource or even extinct. They use this dataset to train and evaluate the largest collection of aligned word embeddings to date, enabling more reliable cross-lingual comparisons and advancing NLP for underrepresented languages.

What Problem Does This Paper Address?

Aligned word embeddings are crucial for many natural language processing (NLP) tasks, especially for low-resource languages where parallel data is scarce. However, existing evaluation datasets for this task, like the MUSE lexicon, are often generated automatically from machine translation systems. This leads to significant errors, including non-existent words and inconsistent coverage, making them unreliable for training and evaluating models. Furthermore, these datasets are typically limited to high-resource languages, hindering research on languages with fewer digital resources. The paper aims to address this by creating a more robust and comprehensive benchmark for evaluating bilingual lexicon induction (BLI) models across a much wider range of languages.

Methods

The core of the paper's contribution lies in its novel data collection and evaluation methodology.

  • Data Extraction: The authors leverage Wiktionary, a collaboratively edited online dictionary, to extract bilingual lexicons. They parse Wiktionary's MediaWiki Markdown language to obtain words, their associated languages, parts of speech, and English definitions.
  • Lexicon Construction: They construct bilingual lexicons for 298 languages into English. A key aspect is the focus on high-quality, human-verified translations, contrasting with automatically generated datasets.
  • Train/Test Splits: To ensure fair cross-lingual comparisons, they develop a language-agnostic approach to splitting data into training and testing sets. They create two test set sizes: a "full test set" of 1500 words and a "small test set" of 250 words. Crucially, they ensure that each test set maintains a consistent number of words for each part of speech (e.g., Adjective, Noun, Verb) across languages, using semantic parts of speech for the smaller set.
  • Model Training and Evaluation: They train bilingual lexicon induction (BLI) models using standard algorithms like VecMap. Their evaluation focuses on the accuracy of aligning word vectors between source languages and English.

Datasets

  • Wiktionary Bilingual Lexicon Collection (New): This is the primary contribution. It provides high-quality, human-annotated translations for words in 298 languages into English. This dataset is publicly available.
  • MUSE Lexicon Collection (Existing): Used for comparison. It covers 45 languages and is generated via machine translation, known for its errors and limitations.
  • Grave et al. (2018) Word Vectors: Pre-trained word vectors for 157 languages, used as the source for training BLI models in their experiments.

Key Insights and Ideas

  • Wiktionary as a High-Quality Source: Wiktionary, despite its collaborative nature, provides a significantly more reliable source for bilingual lexicons than machine-translated datasets, especially for low-resource languages. The human-verified nature of its entries is paramount.
  • Importance of Part-of-Speech Balancing: Ensuring consistent part-of-speech distribution in test sets is critical for fair cross-lingual comparisons. This methodology allows for more meaningful evaluation across languages with varying linguistic structures.
  • Low-Resource Languages Benefit Most: The new Wiktionary dataset opens up the possibility of training and evaluating BLI models for hundreds of languages that previously lacked such resources, enabling research on a much broader linguistic spectrum.
  • Quality of Monolingual Vectors Matters: The paper highlights that even with a good bilingual lexicon, the quality of the underlying monolingual word vectors (trained on corpora like Common Crawl) significantly impacts BLI performance. Languages with fewer web resources (e.g., Latin, Sanskrit) suffer from lower-quality word vectors, hindering alignment despite having rich Wiktionary lexicons.

Key Results

  • Largest BLI Dataset: The paper introduces the largest published collection of aligned word embeddings, covering 157 languages and English.
  • New Datasets for 253 Languages: The Wiktionary dataset provides the first BLI datasets for 253 languages.
  • Improved Quality: The Wiktionary lexicons are shown to be of higher quality than existing datasets like MUSE, leading to more reliable evaluations.
  • Performance on Low-Resource Languages: The paper identifies 15 languages (e.g., Armenian, Georgian, Basque) that show particularly good performance, indicating their suitability for downstream tasks.
  • Dataset Size vs. Performance: Experiments show that BLI accuracy rapidly increases with training vocabulary size up to about 5k words, with diminishing returns thereafter. This suggests that even smaller, high-quality datasets can yield strong results if the monolingual vectors are good.

Limitations and Future Work

  • Computational Resources: The authors acknowledge that they could not train all possible supervised and unsupervised methods due to computational constraints.
  • Quality of Monolingual Vectors: The performance for some languages (e.g., Latin, Sanskrit) is limited by the low quality of their corresponding word vectors trained on Common Crawl, rather than the Wiktionary lexicon itself. This suggests that improving monolingual vector representations for low-resource languages is a crucial area for future work.
  • Dataset Size for Some Languages: While aiming for 1500 words in test sets, some languages have fewer available entries for certain parts of speech, resulting in truncated test sets. The authors suggest that meaningful comparisons can still be made with 80% of a small test set.
  • Future Work: The paper suggests that more data from diverse sources could further improve alignment, and that the new Wiktionary dataset is ideal for enabling better aligned word embeddings for a wider range of downstream applications.

Summary updated

BibTeX
@inproceedings{izbicki2022aligning,
  author = {Mike Izbicki},
  title = {{Aligning Word Vectors on Low-Resource Languages with Wiktionary}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2022},
}
Figure