AREEj: Arabic Relation Extraction with Evidence

Osama Rakan Al Mraikhat, Hadi Hamoud, Fadi A. ZaraketACL 2024

TL;DR

This paper introduces AREEj, a novel model for Arabic relation extraction that incorporates textual evidence to improve accuracy. By augmenting a large Arabic relation extraction dataset with evidence annotations, they demonstrate significant improvements over existing state-of-the-art models, particularly in identifying relations and their supporting evidence.

What Problem Does This Paper Address?

Building comprehensive knowledge graphs requires accurately identifying relationships between entities within text. Traditional relation extraction (RE) models often struggle with ambiguity and lack interpretability. This paper tackles the challenge of enhancing Arabic RE by introducing "evidence" – the specific text snippets that justify a detected relation. This evidence not only helps explain why a model made a certain prediction (contributing to explainable AI) but also acts as a crucial hint, boosting the model's performance. Existing Arabic RE datasets are limited, and models often lack the ability to pinpoint the textual justification for their findings.

Methods

The core of this work involves two main stages:

  1. Evidence Augmentation: The authors first augment the existing SREDFM (a large, multilingual relation extraction dataset) with evidence annotations for its Arabic subset. They achieve this by:
  2. Constructing a Representative Document and Relations Subset (RDRS): They select a subset of Arabic SREDFM documents and relations, focusing on shorter texts to optimize performance with large language models (LLMs).
  3. Base Evidence Annotations: They leverage ChatGPT-4 with a Chain-of-Thought (CoT) prompting strategy to automatically extract evidence for the RDRS relations. This process was optimized to generate evidence or mark it as "none."
    • Fine-tuning an LLM for Evidence: They fine-tune an mBART-based model (named ArBartEv) on these base evidence annotations. This model is designed to take an Arabic text and a relation tuple (source, target, relation type) and output the supporting evidence.
  4. Training AREEj: The fine-tuned ArBartEv model is then used to generate evidence for the entire 2.9-million relation Arabic subset of SREDFM, creating the ArSRED dataset. Finally, they train their proposed model, AREEj, on this augmented ArSRED dataset. AREEj is also based on the mBART architecture and is trained to take Arabic text and produce a relation with its supporting evidence.

Datasets

  • SREDFM (Source, Relation, Entity, Document, Fine-grained, Multilingual): A large, multilingual RE dataset used as the base. The paper focuses on its Arabic subset, which contains 2.9 million relations.
  • ArSRED (Arabic Relation Extraction with Seed Evidence): A newly constructed dataset by augmenting the Arabic SREDFM subset with evidence annotations. This dataset is made publicly available.
  • Testing Set: A manually revised subset of REDFM (a filtered and multilingual RE dataset) containing 345 documents and 864 relations, used for evaluation.

Key Insights and Ideas

  • Evidence as a Performance Booster: The paper strongly supports the idea that providing textual evidence significantly improves relation extraction performance. It acts as a "hint" for the model.
  • Chain-of-Thought for Annotation: Using CoT prompting with ChatGPT-4 was crucial for effectively generating high-quality evidence annotations, demonstrating the power of LLMs for data augmentation in specialized NLP tasks.
  • Fine-tuning for Domain-Specific Tasks: Fine-tuning a pre-trained LLM (mBART) on a domain-specific augmented dataset (ArSRED) leads to a specialized model (ArBartEv) capable of high-quality evidence extraction.
  • AREEj Outperforms SOTA: The proposed AREEj model, trained on the evidence-rich ArSRED dataset, shows superior performance compared to the state-of-the-art mREBEL model, especially when considering both relation extraction and evidence provision.
  • Explainable AI Potential: The inclusion of evidence directly contributes to the explainability of RE models, making their predictions more trustworthy.

Key Results

  • Evidence Augmentation Model (ArBartEv): Achieved an F1-score of .82 for evidence extraction (.93 precision, .73 recall).
  • AREEj Model:
    • Achieved an F1-score of .72 (precision .78, recall .66) for relation extraction with evidence.
  • Outperformed the SOTA mREBEL model, which achieved .72 F1-score (.78 precision, .66 recall) but without evidence.
  • AREEj correctly identifies 72 relations that mREBEL misses, while mREBEL identifies 47 relations AREEj misses.
  • AREEj provides correct evidence .86% of the time.
  • AREEj outperforms mREBEL in relation classification (detecting any relation between entities) by 14 relation types.

Limitations and Future Work

  • Limited Testing Dataset: The testing dataset only covers 32 relation types, which is an initial limitation of the REDFM dataset itself.
  • Future Work:
    • Augmenting REDFM with instances covering all 393 relations.
  • Using extracted evidence to improve precision by relating it to entity and relation types.
  • Reducing the number of "none" (no evidence found) samples to improve recall.

Summary updated

BibTeX
@inproceedings{mraikhat2024areej,
  author = {Osama Rakan Al Mraikhat and Hadi Hamoud and Fadi A. Zaraket},
  title = {{AREEj: Arabic Relation Extraction with Evidence}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure