LU-BZU at SemEval-2021 Task 2: Word2Vec and Lemma2Vec performance in Arabic Word-in-Context disambiguation

Moustafa Al-Hajj, Mustafa JarrarACL 2021

TL;DR

This paper explores the effectiveness of word embeddings (Word2Vec) versus lemma embeddings (Lemma2Vec) for Arabic Word-in-Context (WiC) disambiguation. The study found that while Lemma2Vec slightly outperforms Word2Vec for correctly identifying same-meaning word pairs (TRUE class), Word2Vec is better for pairs with different meanings (FALSE class), leading to comparable overall performance.

What Problem Does This Paper Address?

Disambiguating word senses is crucial for many Natural Language Processing (NLP) tasks like machine translation and information retrieval. Arabic presents unique challenges due to its rich morphology, where a single root can generate thousands of word forms. Traditional word embeddings treat these different forms as distinct words, potentially hindering accuracy. This paper investigates whether using lemma embeddings, which group different word forms under a single lemma, can improve WiC disambiguation performance in Arabic compared to standard word embeddings.

Methods

The researchers employed a system that extracts context for a target word in two sentences, represents these contexts as vectors, and then compares these vectors using cosine similarity to determine if the words have the same meaning. They experimented with two types of embeddings:

  • Word2Vec: Two models were used: one pre-trained on Arabic Wikipedia (Wiki-CBOW) and another trained on a large Arabic corpus (3 billion tokens).
  • Lemma2Vec: These models were derived from the Word2Vec models. For each lemma, the vectors of its associated word forms were averaged to create a single lemma vector.

The system's performance was tuned by experimenting with parameters such as context window size, pooling operations (mean, max, min, std) to combine context word vectors, similarity thresholds, and the removal of functional words (stop words).

Datasets

  • Training/Development: The dev.ar-ar dataset from SemEval-2021 Task 2 was used for tuning parameters and evaluating performance. This dataset consists of 1000 sentence pairs with revealed TRUE/FALSE labels.
  • Testing: The test.ar-ar dataset from SemEval-2021 Task 2 was used for final evaluation. This dataset also contains 1000 sentence pairs but with hidden labels.

Key Insights and Ideas

  • Morphology Matters: The paper highlights how Arabic's complex morphology can be a double-edged sword for word embeddings. Lemma embeddings, by abstracting away from specific word forms, can capture broader semantic similarities, especially when word forms are inflected versions of the same root.
  • Context Sensitivity Trade-off: Lemma embeddings, while potentially more generalized, might become too general. In cases where subtle morphological differences carry crucial semantic distinctions (e.g., distinguishing between "genus" and "sex" based on word form), Word2Vec's specificity might be more advantageous.
  • TRUE vs. FALSE Class Performance: A significant finding is the differential performance of Lemma2Vec and Word2Vec on the TRUE and FALSE classes. Lemma2Vec excels at identifying when words do have the same meaning, while Word2Vec is better at identifying when they don't.

Key Results

  • Overall Performance: The overall accuracy on the test set was comparable between Lemma2Vec and Word2Vec models, with the best performing models achieving around 59-60% accuracy.
  • Lemma2Vec Advantage for TRUE: Lemma2Vec models showed a significant improvement over Word2Vec for the TRUE class (correctly identifying same meanings). For example, in one experiment, 370 pairs were correctly classified by Lemma2Vec vs. Word2Vec, while only 233 were correctly classified by Word2Vec vs. Lemma2Vec.
  • Word2Vec Advantage for FALSE: Conversely, Word2Vec performed better for the FALSE class (correctly identifying different meanings).
  • Parameter Tuning: The study found that shorter context sizes generally benefited Lemma2Vec, while min pooling was often the best operation for combining context vectors. The impact of removing stop words was found to be minor.

Limitations and Future Work

  • Limitations: The paper acknowledges that while Lemma2Vec outperforms Word2Vec for the TRUE class, it underperforms for the FALSE class. Given the balanced distribution of TRUE and FALSE pairs in the dataset, the overall performance remains similar. The authors also note that their approach does not use external resources or context/sense embedding models.
  • Future Work: The researchers plan to extend their work by building multi-prototype embeddings using a large Arabic lexicographic database. They also intend to fine-tune Arabic BERT models on this database, aiming to leverage more advanced contextualized embeddings.

Summary updated

BibTeX
@inproceedings{alhajj2021lu,
  author = {Moustafa Al-Hajj and Mustafa Jarrar},
  title = {{LU-BZU at SemEval-2021 Task 2: Word2Vec and Lemma2Vec performance in Arabic Word-in-Context disambiguation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure