ArabicNLU 2024: The First Arabic Natural Language Understanding Shared Task

Mohammed Khalilia, Sanad Malaysha, Reem Suwaileh, Mustafa Jarrar, Alaa Aljabari, Tamer Elsayed, Imed Zitouni

TL;DR

The ArabicNLU 2024 shared task marks a significant milestone as the first competition dedicated to Word Sense Disambiguation (WSD) and Location Mention Disambiguation (LMD) in Arabic. By introducing two high-quality, manually annotated datasets—SALMA and IDRISI-DA—the authors provide the community with essential benchmarks to tackle the unique morphological and semantic complexities of the Arabic language.

What Problem Does This Paper Address?

Arabic is a linguistically rich language characterized by complex morphology and inherent polysemy (words having multiple meanings). A single word can have vastly different interpretations depending on its context, a challenge magnified by the lack of diacritics in most written text. For example, the word "springs" (‘uyūn) in Arabic can refer to water sources, eyes, or even spies.

While NLU tasks like WSD and LMD are well-studied in English, Arabic has historically suffered from a "data desert." Without large-scale, sense-annotated corpora, it is difficult to build and evaluate models that can truly "understand" Arabic text. This paper addresses this gap by organizing a shared task to stimulate research and providing the necessary datasets to benchmark progress.

Methods

The shared task was divided into two subtasks:

  1. Word Sense Disambiguation (WSD): Determining the correct semantic meaning of a word from a predefined list of candidate senses.
  2. Location Mention Disambiguation (LMD): A specialized form of WSD focused on mapping a location mention (toponym) to a specific geographical entity in a gazetteer like OpenStreetMap (OSM).
Figure 2: Illustration of the WSD subtask.

Baselines and Participant Approaches

The organizers provided strong baselines:

  • WSD Baseline: A BERT-based system using Target Sense Verification (TSV). It treats WSD as a binary classification task, asking the model if a specific "context-gloss" pair is True or False.
  • LMD Baseline: A simple lookup using Nominatim, the official OpenStreetMap search engine.

Participants explored a variety of modern techniques, including:

  • Zero-shot LLMs: Teams used Llama 3 and GPT-4 with carefully crafted prompts to select the correct sense without specific task training.
  • Fine-tuning Encoders: Fine-tuning Arabic-specific models like AraBERTv2 and MARBERT for sequence classification.
  • Retrieval-Reranking: For LMD, one team used a two-stage approach: retrieving candidates from OSM and then reranking them using a multilingual Cohere reranker.

Datasets

The core contribution of this work lies in the two novel datasets provided to the research community:

  • SALMA Corpus (for WSD): The first sense-annotated Arabic corpus. It contains approximately 34k tokens (1,440 sentences) manually annotated with 4,151 unique senses extracted from modern Arabic lexicons.
  • IDRISI-DA (for LMD): The first manually labeled Arabic LMD dataset. It consists of 3,893 location mentions across 2,869 social media posts (X/Twitter) related to disaster events, covering various dialects and seven different countries.
Figure 1: ArabicNLU datasets for WSD and LMD.

Key Insights and Ideas

  • The LLM Hurdle: Interestingly, none of the participating teams—even those using powerful models like Llama 3-70B—were able to beat the organizers' BERT-based TSV baseline (84.2% F1) for WSD. This suggests that for niche, high-precision tasks in lower-resource languages, a smaller encoder model fine-tuned on high-quality data can still outperform a massive generative model in a zero-shot setting.
  • The Translation Workaround: The winning LMD team achieved a staggering 95% MRR@1 by translating Arabic location mentions into English before querying geographical tools like GeoPy. This highlights a significant "performance gap" in native Arabic geospatial tools, forcing researchers to rely on translation as a bridge.
  • Complexity Breeds Caution: Despite 38 teams registering, only three submitted final results. This underscores the sheer difficulty of Arabic NLU and the non-trivial nature of disambiguation tasks compared to simpler tasks like sentiment analysis.

Key Results

The competition results revealed a clear contrast between the two subtasks:

Subtask Best Participant Score Baseline Score Top Team / Method
WSD (F1-score) 77.8% 84.2% Upaya (Llama 3 Zero-shot)
LMD (MRR@1) 94.97% 57.24% Rematchka (Llama 3 + GeoPy)

For LMD, the distribution of location types in the IDRISI-DA dataset (shown below) highlights the diversity of the challenge, ranging from broad countries to specific neighborhoods and human-made points of interest (POIs).

Figure 4: Distribution of location types in IDRISI-DA.

Limitations and Future Work

The authors acknowledge several constraints that pave the way for future research:

  • Dialectal Gap: The SALMA dataset is currently limited to Modern Standard Arabic (MSA). Given that much of the web's Arabic content is dialectal, expanding sense-annotation to dialects is a critical next step.
  • Granularity: The LMD task currently underrepresents fine-grained or temporary locations (like specific street corners or temporary shelters), which are vital during emergency responses.
  • LLM Multilinguality: The poor zero-shot performance of LLMs on WSD is partly attributed to the fact that only a small fraction (e.g., 5% for Llama 3) of their training data is multilingual. Improving the representation of Arabic in these models is essential for better "out-of-the-box" NLU.

Summary updated

Abstract

This paper presents an overview of the Arabic Natural Language Understanding (ArabicNLU 2024) shared task, focusing on two subtasks: Word Sense Disambiguation (WSD) and Location Mention Disambiguation (LMD). The task aimed to evaluate the ability of automated systems to resolve word ambiguity and identify locations mentioned in Arabic text. We provided participants with novel datasets, including a sense-annotated corpus for WSD, called SALMA with approximately 34k annotated tokens, and the IDRISI-DA dataset with 3,893 annotations and 763 unique location mentions. These are challenging tasks. Out of the 38 registered teams, only three teams participated in the final evaluation phase, with the highest accuracy being 77.8% for WSD and the highest MRR@1 being 95.0% for LMD. The shared task not only facilitated the evaluation and comparison of different techniques, but also provided valuable insights and resources for the continued advancement of Arabic NLU technologies.
BibTeX
@inproceedings{khalilia2024arabicnlu,
  author = {Mohammed Khalilia and Sanad Malaysha and Reem Suwaileh and Mustafa Jarrar and Alaa Aljabari and Tamer Elsayed and Imed Zitouni},
  title = {{ArabicNLU 2024: The First Arabic Natural Language Understanding Shared Task}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure