WojoodRelations: Arabic Relation Extraction Corpus and Modeling

Alaa Aljabari, Mohammed Khalilia, Mustafa JarrarEMNLP 2025

TL;DR

This paper introduces WojoodRelations, the largest and most diverse Arabic relation extraction (RE) corpus to date, featuring over 33,000 sentences and 40 relation types. They also propose NLI-RE, a novel framework that frames RE as a natural language inference problem, and GPT-Joint, a few-shot LLM approach for joint entity and relation extraction, demonstrating strong baselines for Arabic RE.

What Problem Does This Paper Address?

Extracting structured information from text, known as Relation Extraction (RE), is crucial for understanding natural language and building knowledge graphs. However, Arabic RE faces significant hurdles due to the language's complex morphology and syntax, coupled with a scarcity of large, high-quality annotated datasets. Existing Arabic RE resources are often limited in size, scope, or are not publicly available, hindering research progress. This paper aims to bridge this gap by creating a comprehensive Arabic RE corpus and developing effective modeling approaches.

Methods

The authors introduce two main methodological contributions:

  • NLI-RE (Natural Language Inference for Relation Extraction): This novel framework reframes RE as a binary natural language inference task. It uses relation-aware templates to explicitly condition the inference on specific relation types, allowing models to determine if a hypothesized relation between two entities holds true in a given sentence. This approach is designed to be effective even for implicit relations.
  • GPT-Joint: This is a few-shot LLM framework for joint entity and relation extraction. It incorporates a "relation-aware retrieval" mechanism to select semantically relevant demonstrations for the LLM, aiming to improve the extraction of both entities and their relationships simultaneously.

They also evaluate existing methods like graph-based RE (RIFRE) and LLM-based RE (GPT-RE) on their new dataset.

Datasets

The core contribution is the WojoodRelations corpus:

  • Size: Over 33,000 sentences (~550,000 tokens).
  • Annotations: ~15,000 relation triples across 40 relation types.
  • Foundation: Built upon the Wojood NER dataset, with manual annotations by expert annotators achieving high reliability (Cohen's κ of 0.92).
  • Coverage: Includes diverse domains and aims to capture the linguistic complexity of Arabic, including nested entities.

Additionally, they create task-specific datasets derived from WojoodRelations for evaluation:

  • RelGraph: For graph-based RE.
  • RelGPT: For LLM-based RE.
  • RelNLI: For NLI-based RE.
  • RelJoint: For joint extraction tasks.

Key Insights and Ideas

  • The power of NLI for RE: Framing RE as an NLI problem (NLI-RE) proves highly effective, achieving the highest macro F1 score among all tested models. This suggests that explicitly modeling entailment can better capture both explicit and implicit relations.
  • LLMs still face challenges in RE: While LLMs show promise, especially in few-shot settings, they often underperform compared to supervised methods on this corpus, particularly for rare or implicit relations. Their performance can be inconsistent.
  • Data quality is paramount: The high inter-annotator agreement (0.92 Cohen's κ) for WojoodRelations underscores the importance of meticulously curated datasets for advancing Arabic RE.
  • Arabic RE is complex: The paper highlights specific challenges like lexical variation and ambiguous entity mentions, which require careful handling.

Key Results

  • NLI-RE achieved the highest macro F1 score of 88.58%, outperforming other models in generalizing over relation types.
  • Supervised models (like NLI-RE and RIFRE) generally outperformed LLM-based approaches on the WojoodRelations corpus, with supervised models achieving up to 92.89% F1 for relation extraction.
  • LLMs in few-shot settings (like GPT-Joint) achieved 72.73% F1 for joint entity and relation extraction, showing improvement over zero-shot LLM approaches.
  • RIFRE (graph-based) showed strong performance on high-frequency relations but struggled with rare and implicit ones.
  • GPT-RE (LLM-based) showed improved performance on long-tail relations compared to RIFRE but was still limited by its reliance on surface-level patterns.

Limitations and Future Work

  • MSA Focus: The WojoodRelations corpus primarily covers Modern Standard Arabic (MSA) within formal domains, with limited representation of informal, dialectal, or conversational Arabic. This restricts the applicability of models trained on this corpus to non-MSA varieties.
  • Computational Overhead: The NLI-RE approach, while effective, requires inference over all templates for each entity pair, leading to some computational overhead.
  • Rare Relations: Detecting implicit and rare relations remains a challenge.
  • Future Work: The authors suggest improving generalization to long-tail relations and enhancing the use of contextual information to advance Arabic RE. They also propose exploring data augmentation and few-shot learning strategies for handling rare relation types.

Summary updated

BibTeX
@inproceedings{aljabari2025wojoodrelations,
  author = {Alaa Aljabari and Mohammed Khalilia and Mustafa Jarrar},
  title = {{WojoodRelations: Arabic Relation Extraction Corpus and Modeling}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure