Pirates at ArabicNLU2024: Enhancing Arabic Word Sense Disambiguation using Transformer-Based Approaches

Tasneem Wael, Eman Elrefai, Mohamed Makram, Sahar Selim, Ghada KhoribaACL 2024

TL;DR

This paper tackles the challenging task of Arabic Word Sense Disambiguation (WSD) by leveraging advanced transformer-based models. They achieved impressive results, securing second place in the ArabicNLU 2024 competition, by adapting techniques like Sentence Transformers and the SetFit framework for few-shot learning, demonstrating their effectiveness on complex Arabic linguistic features.

What Problem Does This Paper Address?

Arabic is a morphologically rich language, making computational linguistics tasks like Word Sense Disambiguation (WSD) particularly difficult. The presence of diacritics, prefixes, and suffixes, along with the common absence of diacritics in written texts, leads to numerous homophones and homographs. This ambiguity means a single word can have multiple meanings depending on its context, making it hard for machines to accurately determine the intended sense. Traditional WSD methods often struggle with these complexities, especially with limited or noisy data. This paper aims to improve Arabic WSD by exploring and adapting state-of-the-art transformer models.

Methods

The researchers explored three main transformer-based approaches:

  • Sentence Transformer with Siamese Networks: This method fine-tuned the AraBERTv2 model using a contrastive loss. The core idea was to train the model to embed sentences and word senses into a shared space, then use the Euclidean distance between these embeddings to distinguish between matching and non-matching senses. A logistic regression model was used to find an optimal distance threshold.
  • SetFit (Few-Shot Learning): Inspired by computer vision techniques, SetFit was adapted for WSD. This framework optimizes model performance by learning to generalize from a small number of labeled examples (few-shot learning). It leverages Sentence Transformers but in a more optimized pipeline, making it suitable for datasets with limited data.
  • Classification Approach: This is a more standard approach where transformer models are used directly for classification tasks. The authors fine-tuned the pre-trained AraBERTv2 model with a single linear classification layer. This method is described as commonly used and stable.

Datasets

  • Combined Dataset: This dataset was constructed by merging two resources:
  • El Gani: A large Arabic dictionary containing over 527,000 samples.
  • Arabic Context Gloss pairs: A dataset with over 310,000 samples. This combined dataset was balanced and used for the first experiment.

  • SALMA Dataset: The development and test sets of the SALMA dataset were used for training and evaluation in the second and third experiments. This dataset is described as having a similar distribution to the test set, which is expected to yield higher F1-scores.

Key Insights and Ideas

  • Transformer Models Excel at Arabic WSD: The study strongly suggests that transformer-based models, particularly AraBERTv2, are highly effective for Arabic WSD, significantly outperforming traditional methods.
  • Few-Shot Learning Shows Promise: The SetFit framework, designed for few-shot learning, demonstrated its potential for Arabic WSD, even though it's typically used for other tasks like intent classification. This highlights its adaptability to low-resource scenarios.
  • Overfitting is a Concern: While the classification approach yielded the best results, the authors noted signs of overfitting, indicating that generalization remains a key challenge.
  • Dataset Distribution Matters: The discrepancy in performance between the combined dataset test set and the SALMA test set for the Sentence Transformer approach suggests that matching the training data distribution to the test data distribution is crucial for better generalization.

Key Results

The paper reports the following F1-scores on the SALMA test set:

  • Experiment 1 (Sentence Transformer): 48% F1-score.
  • Experiment 2 (SetFit): 50% F1-score.
  • Experiment 3 (Classification Approach): 71% F1-score.

The classification approach achieved the highest F1-score of 71% on the SALMA test set and secured second place in the ArabicNLU 2024 competition. The same approach achieved a 94% F1-score on the development data, further indicating potential overfitting.

Limitations and Future Work

  • Overfitting: The authors acknowledge that the best-performing classification approach showed signs of overfitting, suggesting the need for improved generalization.
  • Dataset Distribution Impact: Future work will involve deeper investigations into how dataset distribution affects model performance and generalization.
  • SetFit Optimization: Further exploration of SetFit is planned, including investigating different loss functions and hyperparameter tuning to optimize it specifically for WSD.
  • Data Augmentation: The researchers aim to enhance data augmentation techniques to create more diverse training data and improve model generalization.
  • Hybrid Approaches: Combining multiple techniques and incorporating strategies like verb preposition classes are also proposed for future research.

Summary updated

BibTeX
@inproceedings{wael2024pirates,
  author = {Tasneem Wael and Eman Elrefai and Mohamed Makram and Sahar Selim and Ghada Khoriba},
  title = {{Pirates at ArabicNLU2024: Enhancing Arabic Word Sense Disambiguation using Transformer-Based Approaches}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure