Evaluating a Joint Training Approach for Learning Cross-lingual Embeddings with Sub-word Information without Parallel Corpora on Lower-resource Languages

Ali Hakimi Parizi, Paul CookACL 2021

TL;DR

This paper evaluates a novel method for learning cross-lingual word embeddings that leverages sub-word information, making it particularly effective for low-resource languages. By incorporating sub-word units, the approach significantly improves performance on out-of-vocabulary (OOV) words, a common challenge in morphologically rich languages.

What Problem Does This Paper Address?

Cross-lingual word embeddings are crucial for transferring knowledge between languages in NLP tasks. Traditional methods often rely on parallel corpora (texts aligned sentence-by-sentence), which are scarce for many of the world's languages. While unsupervised methods have emerged, they often struggle with morphologically rich languages and out-of-vocabulary (OOV) words. This paper tackles the challenge of creating effective cross-lingual embeddings for low-resource languages without relying on parallel data, specifically addressing the OOV problem by incorporating sub-word information.

Methods

The core of this work is an evaluation of the HAKIMI2020 method, which extends a joint training approach (Duong et al., 2016). The key innovation is the incorporation of sub-word information during the joint training process. Instead of treating words as atomic units, HAKIMI2020 breaks words down into character n-grams (typically of length 3-6), including special start and end markers. This allows the model to learn representations for words it hasn't seen before by composing them from known sub-word units. The training process uses monolingual corpora and a bilingual dictionary as a cross-lingual signal, avoiding the need for parallel data.

Datasets

The study uses Wikipedia dumps from July 2020 as training corpora for twelve lower-resource languages and English. For evaluation, they utilize bilingual lexicon induction (BLI) test data. For in-vocabulary (IV) words, they use MUSE test data (except for Azerbaijani, which uses data from Anastasopoulos and Neubig, 2020). For out-of-vocabulary (OOV) words, they construct test sets using Panlex, focusing on translation pairs where the source word is OOV with respect to the learned embeddings.

Key Insights and Ideas

  • Sub-word information is key for OOVs: The most significant finding is that incorporating sub-word information dramatically improves performance on out-of-vocabulary words, a critical limitation of previous word-level methods.
  • Joint training shines for low-resource languages: The HAKIMI2020 method, which jointly trains embeddings and uses sub-word information, proves to be a strong contender for low-resource scenarios where parallel data is unavailable.
  • Combining approaches can be beneficial: The paper introduces a hybrid approach (HAKIMI2020+COPY) that combines the HAKIMI2020 method with a simple "copy" strategy for named entities or borrowings, leading to further improvements for OOV words.
  • Language families matter: While not fully explored, the authors suggest future work on understanding the impact of source and target languages belonging to the same family.

Key Results

The paper presents results for Bilingual Lexicon Induction (BLI) on twelve lower-resource languages.

  • In-Vocabulary (IV) Words: HAKIMI2020 shows improvements over the DUONG2016 baseline. While MEEMI performs better at precision@1, HAKIMI2020 outperforms MEEMI at precision@10.
  • Out-of-Vocabulary (OOV) Words: HAKIMI2020 significantly outperforms the VECMAP baseline across all languages and evaluation measures. The proposed HAKIMI2020+COPY approach further boosts performance, achieving substantial improvements on average for OOV words. For instance, for Afrikaans, HAKIMI2020+COPY achieves a precision@1 of 19.16, compared to VECMAP's 5.65.

Limitations and Future Work

  • Limitations: The authors acknowledge that the Panlex translations used for OOV evaluation can sometimes be noisy. They also note that while HAKIMI2020+COPY generally improves performance, there are specific languages (like Bengali and Hindi) where the improvement is minimal, potentially due to the simple "COPY" strategy not being as effective.
  • Future Work: The authors plan to explore the impact of the target language on cross-lingual embedding quality, particularly focusing on source and target languages from the same family. They also intend to evaluate these embeddings in downstream tasks for low-resource languages, such as language modeling and part-of-speech tagging, and compare them with approaches based on contextualized language models.

Summary updated

BibTeX
@inproceedings{parizi2021evaluating,
  author = {Ali Hakimi Parizi and Paul Cook},
  title = {{Evaluating a Joint Training Approach for Learning Cross-lingual Embeddings with Sub-word Information without Parallel Corpora on Lower-resource Languages}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure