NPVec1: Word Embeddings for Nepali - Construction and Evaluation

Pravesh Koirala, Nobal B. NiraulaACL 2021

TL;DR

This paper introduces NPVec1, the largest and most comprehensive set of pre-trained word embeddings for Nepali, a low-resource language. By training on a massive corpus and evaluating various preprocessing techniques, the authors provide valuable resources to accelerate NLP research and applications for Nepali.

What Problem Does This Paper Address?

Natural Language Processing (NLP) has seen tremendous advancements thanks to deep learning techniques. A crucial component of these techniques is word embeddings, which represent words as dense vectors capturing semantic meaning. However, developing effective word embeddings for low-resource languages like Nepali is challenging due to the scarcity of large, high-quality text corpora. Furthermore, Nepali's morphologically rich nature, with its agglutinative suffixes and varied written forms, complicates preprocessing steps like tokenization and normalization, which are vital for accurate embedding generation. This paper aims to address these challenges by constructing and evaluating a large-scale set of Nepali word embeddings.

Methods

The authors employed several state-of-the-art word embedding models:

  • Context-Independent Embeddings: They utilized GloVe, Word2Vec, and fastText. These models were trained using the skip-gram method (for Word2Vec and fastText) and a custom tool (for GloVe).
  • Context-Dependent Embeddings: They trained a BERT model. Due to resource constraints, they used a simplified BERT architecture (6 attention heads, 6 hidden layers, 300 dimensions) and trained it for only one epoch on the Masked Language Modelling (MLM) objective.

A significant aspect of their methodology was the systematic exploration of preprocessing techniques. They investigated eight different combinations of:

  • Base: Raw text.
  • Normalization (N): Standardizing different written forms of vowels and mapping them to a single representation.
  • Tokenization (T): Breaking down words with post-positions and agglutinative suffixes into their constituent parts.
  • Stemming (S): Reducing inflected verb forms to their base forms.

These preprocessing schemes were applied to create multiple versions of the corpus for training the embeddings.

Datasets

The corpus for training the word embeddings was constructed by combining data from several sources:

  • News Corpus: Over 700,000 Nepali online news articles (~3GB) crawled over a year, supplemented by a 1.7GB dataset from IEEE DataPort.
  • OSCAR Nepali Corpus: A deduplicated subset (1.2GB) of the Open Super-large Crawled ALMAnaCH coRpus.
  • Nepali Wikipedia Corpus: 39,000 Wikipedia articles for Nepali (83MB) obtained from Kaggle.

For evaluation, the authors created two novel datasets:

  • Relatedness Set: Consisting of 21 word examples each from "kitchen" and "nature" topics.
  • Sentiment Set: Comprising 19 examples each of positive and negative sentiments.

These datasets were used with K-Means clustering to evaluate the quality of the learned embeddings. Additionally, for extrinsic evaluation, they used the Nepali News Dataset, a publicly available repository with news articles across 10 categories.

Key Insights and Ideas

  • Preprocessing Matters Significantly: The study highlights that preprocessing, particularly normalization and tokenization, has a substantial impact on the performance of word embeddings for Nepali. Normalization, by reducing linguistic noise from varied spellings, generally improved performance.
  • Context-Independent Embeddings Excel in Relatedness: GloVe, fastText, and Word2Vec models demonstrated strong performance in capturing word relatedness, often achieving perfect scores in the intrinsic evaluation.
  • BERT's Potential for Sentiment: While BERT didn't outperform other models on the relatedness task, it showed competitive results on the sentiment task, suggesting its potential for sentiment analysis applications.
  • Stemming Can Be Detrimental: The inclusion of stemming in preprocessing schemes often led to a drop in performance across various tasks. This suggests that over-stemming might be an issue for Nepali, potentially losing valuable semantic nuances.
  • Resource Constraints Impact BERT: The authors acknowledge that their simplified BERT model, trained with limited epochs and resources, likely underperformed compared to a full-fledged BERT implementation.

Key Results

  • Largest Embeddings for Nepali: NPVec1 comprises 25 word embedding models, claimed to be the largest ever trained for Nepali.
  • Intrinsic Evaluation: Models trained with normalization and tokenization (BNT scheme) generally performed best in intrinsic evaluations. GloVe achieved the highest global score (0.69) in the sentiment set.
  • Extrinsic Evaluation: Pre-trained word embeddings consistently improved the performance of a text classification model compared to a baseline model that learned embeddings from scratch. On average, F1 scores for word embedding models exceeded baseline scores by 5%. The "Base" scheme (no preprocessing) yielded the highest global F1 score in extrinsic evaluation, possibly due to the clean nature of the news data.
  • FastText's Resilience: FastText showed more resilience to preprocessing variations, with a lower standard deviation in F-scores across different schemes.

Limitations and Future Work

  • Stemming Issues: The paper notes that stemming can negatively impact performance, suggesting the need for improved stemmers or further investigation into its effects.
  • BERT's Limited Training: The authors acknowledge that their BERT model was trained with significantly fewer resources (360 million words) and epochs than standard implementations (3.3 billion words), limiting its potential. They recommend training a full BERT architecture for better results.
  • Exploration of Other Models: Future work could involve evaluating other context-dependent embedding models like ELMo, GPT2, XLNet, and RoBERTa for Nepali.
  • Task-Specific Embeddings: The authors suggest exploring the performance of these embeddings in tasks like Named Entity Recognition and POS Tagging.
  • Improved Stemmer: Further comparisons with an improved stemmer that has fewer over-stemming errors are recommended.

Summary updated

BibTeX
@inproceedings{koirala2021npvec,
  author = {Pravesh Koirala and Nobal B. Niraula},
  title = {{NPVec1: Word Embeddings for Nepali - Construction and Evaluation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure