Toward Better Loanword Identification in Uyghur Using Cross-lingual Word Embeddings

Chenggang Mi, Yating Yang, Lei Wang, Xi Zhou, Tonghai JiangCOLING 2018 · OpenReview

TL;DR

This paper tackles the challenge of identifying loanwords in low-resource languages like Uyghur by leveraging cross-lingual word embeddings. Their novel approach combines semantic similarity from embeddings with phonetic and language model features to accurately detect loanwords, significantly improving both loanword identification and subsequent machine translation of out-of-vocabulary (OOV) words.

What Problem Does This Paper Address?

Many natural language processing (NLP) tasks, especially for low-resource languages, suffer from data sparsity. This is particularly problematic for tasks like machine translation where a significant portion of words might be "out-of-vocabulary" (OOV). Loanwords, words adopted from one language into another, are a common source of OOV words. Identifying these loanwords is crucial for enriching the vocabulary of low-resource languages and improving NLP applications like machine translation. Previous methods often relied on rule-based approaches or machine learning models that struggled with data scarcity and semantic ambiguity. This paper aims to address this by proposing a robust loanword identification method for Uyghur, a low-resource language with significant historical and ongoing linguistic influences from various donor languages.

Methods

The proposed method has two main stages:

  1. Loanword Candidate List Generation:
    • Cross-lingual Word Embeddings (CdbCLE): The core of this stage is training cross-lingual word embeddings using comparable corpora (Uyghur and a donor language) and a small bilingual lexicon. This method, inspired by existing work, creates a joint embedding space where semantically similar words from different languages are close together.
  2. Semantic Similarity: Given a Uyghur word, the model finds its most semantically similar source words in the donor language using the trained embeddings.
  3. Candidate Filtering: A threshold p is used to filter out Uyghur words whose closest donor language match is too distant semantically, assuming these are unlikely to be loanwords or have significantly changed meaning. Part-of-speech tagging is also used to filter candidates, as loanwords in Uyghur are often nouns.
  4. Loanword Prediction Model:
    • Log-linear Model: A log-linear model is employed to predict whether a candidate word is a loanword or not. This model combines several features:
  5. Transliteration: To handle languages with different writing systems, a transliteration model is used to create a character mapping table, allowing for direct comparison of phonetic features.
  6. Pronunciation Similarity: This feature uses a position-related edit distance (PRED) method to measure the phonetic similarity between the Uyghur word and its potential donor word, accounting for potential suffix differences.
  7. Hybrid Language Model: This feature combines character-level language models from both the receipt (Uyghur) and donor languages to capture the pronunciation of the loanword.

Datasets

The paper utilizes two types of datasets:

  • Comparable Corpora:
    • BilCorp4MT: Bilingual corpora for training and developing NMT models, including Uyghur-Arabic, Uyghur-Chinese, Uyghur-Russian, and Uyghur-Persian pairs. These were collected from various sources and augmented with manually generated bilingual lexicons. The test set for this corpus was manually annotated for loanword information.
  • CompCorp4Train: These datasets (UyghurMono, ChnMono, AraMono, RusMono, PerMono) were used to train word embedding models and for cross-lingual word embedding-based candidate generation. Some loanword pairs (100) were extracted from this set to train transliteration models.

Key Insights and Ideas

  • Bridging the Low-Resource Gap: Cross-lingual word embeddings are a powerful tool for bridging the gap in low-resource settings, enabling semantic comparisons even without extensive parallel data.
  • Multi-faceted Loanword Detection: Loanword identification is not just about semantic similarity; incorporating phonetic and language model features significantly improves accuracy, especially when dealing with different writing systems.
  • Transliteration as a Preprocessing Step: For languages with different scripts, a transliteration model is crucial for enabling phonetic comparisons, making the system more versatile.
  • The Importance of Candidate Generation: Generating a high-quality list of potential loanwords using semantic similarity is a critical first step before applying more complex prediction models.
  • Loanwords Enhance NMT: Identifying and incorporating loanwords can directly improve the performance of Neural Machine Translation (NMT) systems, particularly for handling OOV words.

Key Results

  • Loanword Identification: The proposed method significantly outperforms baseline models (MutilShlFeats and NNBased) in loanword identification across all four donor languages (Arabic, Chinese, Persian, Russian). For example, in Arabic, the F1-score reached 77.42% compared to 76.31% for NNBased and 74.94% for MutilShlFeats.
  • OOV Translation: Integrating the identified loanwords into NMT systems led to substantial improvements. The best system (sys3, BPE-based with loanword pairs in post-processing) achieved BLEU score improvements of 0.5-0.9 points over the baseline across various language pairs. OOV rates also decreased significantly.
  • Threshold Sensitivity: The precision of the loanword identification model is sensitive to the semantic similarity threshold p, with optimal values varying slightly across donor languages, suggesting language-specific tuning might be beneficial.

Limitations and Future Work

  • Threshold Tuning: The performance is sensitive to the threshold p, requiring careful tuning for each language pair.
  • Data Sparsity for Russian: The paper notes that loanword identification for Russian was less effective than for Arabic, Persian, and Chinese, potentially due to relatively fewer corpora for Russian.
  • Future Work: The authors plan to explore transfer learning to further enhance their model's performance.

Summary updated

Abstract

To enrich vocabulary of low resource settings, we proposed a novel method which identify loanwords in monolingual corpora. More specifically, we first use cross-lingual word embeddings as the core feature to generate semantically related candidates based on comparable corpora and a small bilingual lexicon; then, a log-linear model which combines several shallow features such as pronunciation similarity and hybrid language model features to predict the final results. In this paper, we use Uyghur as the receipt language and try to detect loanwords in four donor languages: Arabic, Chinese, Persian and Russian. We conduct two groups of experiments to evaluate the effectiveness of our proposed approach: loanword identification and OOV translation in four language pairs and eight translation directions (Uyghur-Arabic, Arabic-Uyghur, Uyghur-Chinese, Chinese-Uyghur, Uyghur-Persian, Persian-Uyghur, Uyghur-Russian, and Russian-Uyghur). Experimental results on loanword identification show that our method outperforms other baseline models significantly. Neural machine translation models integrating results of loanword identification experiments achieve the best results on OOV translation(with 0.5-0.9 BLEU improvements)
BibTeX
@inproceedings{mi2018toward,
  author = {Chenggang Mi and Yating Yang and Lei Wang and Xi Zhou and Tonghai Jiang},
  title = {{Toward Better Loanword Identification in Uyghur Using Cross-lingual Word Embeddings}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=uaYLkN9s3k},
}
Figure