Transferred Embeddings for Igbo Similarity, Analogy, and Diacritic Restoration Tasks

Ignatius Ezeani, Ikechukwu Onyenwe, Mark HeppleCOLING 2018

TL;DR

This paper tackles the challenge of building NLP tools for low-resource languages like Igbo by leveraging transfer learning. They successfully adapted English word embeddings to Igbo for semantic tasks and introduced a novel method for learning diacritic embeddings, significantly improving diacritic restoration performance.

What Problem Does This Paper Address?

Developing Natural Language Processing (NLP) tools for languages with limited digital resources is a significant hurdle. Most state-of-the-art NLP models are trained on massive datasets from well-resourced languages like English. For minority languages, creating similar datasets from scratch is prohibitively expensive and time-consuming. This paper focuses on Igbo, a language spoken by over 30 million people but critically lacking in NLP resources. The authors aim to bridge this gap by applying transfer learning to create effective Igbo NLP models, specifically addressing semantic tasks (word similarity, analogy) and a unique challenge: diacritic restoration, which is crucial for disambiguating meaning in Igbo.

Methods

The core of this research lies in transfer learning for word embeddings. Instead of training models from scratch on limited Igbo data, they project existing English-trained embeddings onto Igbo. This projection is achieved using an alignment-based method (Guo et al., 2015), which leverages bilingual dictionaries to map English word vectors to their Igbo counterparts.

For the diacritic restoration task, they introduce a novel approach: learning diacritic embeddings. This involves creating specialized embeddings for diacritic-marked words by considering the weighted average of their most co-occurring English words. The restoration process then uses cosine similarity between context vectors and candidate diacritic variants to select the most appropriate one.

Datasets

The paper utilizes several datasets:

  • Igbo Bible: A primary source for Igbo text, used for training the igBible embedding model.
  • Igbo Novels and Official Documents: Additional Igbo texts used to augment the training data.
  • Igbo-English Parallel Bible Corpora: Crucial for word alignment and the projection of embeddings.
  • English-trained Embeddings: Various pre-trained English models (e.g., Google News, fastText Wikipedia) were used as sources for transfer learning.
  • Word Similarity Dataset: Transferred from the standard wordsim353 dataset and translated into Igbo using Google Translate.
  • Analogy Dataset: Created by generating pairs of opposites for common Igbo nouns and adjectives.
  • Odd Word Dataset: Auto-generated from Igbo words categorized into simple, mutually exclusive groups.

Key Insights and Ideas

  • Transfer Learning is Effective for Low-Resource Languages: Projected Igbo embeddings significantly outperform models trained on limited Igbo data for semantic tasks like word similarity and analogy. This validates the approach of leveraging knowledge from high-resource languages.
  • Diacritic Embeddings are Crucial for Igbo NLP: The introduction of learned diacritic embeddings dramatically improves performance on the diacritic restoration task, highlighting the importance of handling language-specific orthographic features.
  • News-domain Embeddings Perform Well: Models trained on general news data (e.g., igGNews, igWkNews) generally performed better on semantic tasks compared to those trained on more specialized data like the Igbo Bible.
  • Subword Information's Mixed Success: While subword information (igWkSbwd) is theoretically beneficial for agglutinative languages like Igbo, its effectiveness was limited in this study, possibly due to differences in subword patterns between the source and target languages.

Key Results

  • Semantic Tasks: Projected Igbo embeddings generally outperformed baseline models on odd-word, word similarity, and analogy tasks. For instance, igWkNews achieved an accuracy of 81.51% on the odd-word task, 59.69% on similarity, and 80.95% on analogy.
  • Diacritic Restoration: The "Diac" versions of the models, which incorporated learned diacritic embeddings, showed substantial improvements over their "Basic" counterparts. For example, the igBible model's accuracy increased from 69.28% (Basic) to 82.26% (Diac), and precision from 61.37% to 77.96%. The igGNews model saw similar gains, reaching 74.14% accuracy and 72.50% precision with diacritic embeddings.

Limitations and Future Work

  • Limited Igbo Data: The authors acknowledge that the Igbo data used, even with augmentation, is still relatively small, which can impact model performance.
  • Subword Pattern Mismatch: The subword patterns in the pre-trained models might not align well with Igbo's structure, limiting the effectiveness of models like igWkSbwd.
  • Future Work: The authors plan to refine their techniques and datasets, explore models with subword information more deeply, and investigate sense disambiguation tasks. They also aim to build a comprehensive framework for Igbo NLP as part of the IgboNLP project.

Summary updated

BibTeX
@inproceedings{ezeani2018transferred,
  author = {Ignatius Ezeani and Ikechukwu Onyenwe and Mark Hepple},
  title = {{Transferred Embeddings for Igbo Similarity, Analogy, and Diacritic Restoration Tasks}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
}
Figure