Event-Arguments Extraction Corpus and Modeling using BERT for Arabic

Alaa Aljabari, Lina Duaibes, Mustafa Jarrar, Mohammed KhaliliaACL 2024

TL;DR

This paper introduces the Wojood Hadath corpus, a new annotated dataset for Arabic event-argument extraction, and proposes a novel method that frames this task as a Natural Language Inference (NLI) problem using BERT. Their approach achieves state-of-the-art performance, demonstrating the effectiveness of NLI for this challenging task, especially for low-resource languages like Arabic.

What Problem Does This Paper Address?

Understanding and extracting events from text is crucial for many natural language understanding applications, such as disaster monitoring and information retrieval. Event-argument extraction (EAE) specifically focuses on identifying events and linking them to their associated participants (agents), locations, and dates. However, a significant challenge, particularly for Arabic, is the scarcity of high-quality, annotated linguistic resources. This gap hinders the development and evaluation of robust EAE systems for Arabic. The authors aim to address this by creating a new corpus and developing an effective EAE method for Arabic.

Methods

The core of the paper's methodology lies in reframing the EAE task as a Natural Language Inference (NLI) problem. Instead of directly training a model for EAE, they leverage the power of BERT by treating it as a text entailment task.

Here's a breakdown of their approach:

  • Corpus Creation: They extended the existing Wojood corpus, a large Arabic nested named entity corpus, by manually annotating event-argument relations. They focused on three argument types: agent, location, and date.
  • NLI Framing: For a given sentence (premise), they automatically generate hypotheses that represent potential event-argument relationships using predefined templates. For example, a template might be "[Event] is the agent of [Entity]".
  • BERT as NLI Classifier: A BERT model is then trained to classify whether the premise sentence entails the generated hypothesis. A "True" entailment indicates the presence of the event-argument relationship.
  • Template Construction: They designed specific templates for each of the three relation types (hasAgent, hasLocation, hasDate) to generate hypotheses. These templates are parameterized by event and entity mentions.
  • Weighted Loss Function: To prioritize the accurate identification of positive instances (where a relationship exists), they employed a weighted cross-entropy loss function, giving higher weight to positive examples. They also incorporated Noise Contrastive Estimation (NCE) loss to improve discrimination between positive and negative instances.
  • End-to-End System: Finally, they integrated their trained NLI model into an end-to-end system that first performs Named Entity Recognition (NER) and then uses the NLI model to extract event arguments.

Datasets

The paper introduces and utilizes the following datasets:

  • Wojood Hadath Corpus: This is a newly created corpus, an extension of the existing Wojood corpus. It contains 550k tokens and is manually annotated with event-argument relations (agent, location, date).
  • Hadath NLI Dataset: Derived from the Wojood Hadath corpus, this dataset consists of premise-hypothesis sentence pairs specifically designed for training and evaluating the NLI-based EAE model. It includes both positive (entailment) and negative (non-entailment) pairs.
  • WojoodOutOfDomain Corpus: An additional corpus of approximately 80k tokens, collected from 10 diverse domains. This dataset was used to evaluate the generalization capabilities of their proposed method on unseen domains.
  • WojoodOutOfDomain NLI Dataset: An NLI dataset generated from the WojoodOutOfDomain corpus for testing the model's performance on out-of-domain data.

Key Insights and Ideas

  • NLI as a Powerful Framework for EAE: The paper strongly suggests that framing EAE as an NLI task is a highly effective approach, especially for languages with limited resources. This method leverages the strong generalization capabilities of pre-trained language models like BERT.
  • Importance of Annotated Corpora: The creation of the Wojood Hadath corpus is a significant contribution, providing a much-needed resource for Arabic EAE research. The high inter-annotator agreement (82.23% Kappa, 87.2% F1) indicates the quality and reliability of the annotations.
  • Template-Based Hypothesis Generation: The systematic use of templates to generate hypotheses for the NLI task allows for a structured and scalable way to represent potential event-argument relationships.
  • Weighted Loss for Imbalanced Data: The choice of a weighted loss function is crucial for handling the inherent imbalance in EAE tasks, where negative instances (no relation) often far outweigh positive instances.

Key Results

  • High Performance on Hadath NLI: The proposed BERT-based NLI method achieved an impressive F1-score of 94.01% on the Hadath NLI test set.
  • Strong Generalization: The model demonstrated good generalization capabilities, achieving an F1-score of 83.59% on the out-of-domain WojoodOutOfDomain test set.
  • Inter-Annotator Agreement: The manual annotation process for the Wojood Hadath corpus yielded high agreement scores: 82.23% Kappa and 87.2% F1-score.
  • Relation-Specific Performance: The NLI model showed particularly strong performance for the hasDate relation, achieving an F1-score of 96.44%.

Limitations and Future Work

  • MSA Focus: The constructed corpora (Wojood Hadath and WojoodOutOfDomain) primarily focus on Modern Standard Arabic (MSA) and do not cover dialectal variations.
  • Domain Specificity: While out-of-domain testing was performed, the results are still constrained to the specific domains included in their study.
  • Future Work with LLMs: The authors suggest exploring the capabilities of Large Language Models (LLMs) to further enhance event-argument extraction by improving named entity recognition and relationship extraction for knowledge graph construction. They also envision integrating LLMs to potentially boost accuracy and scalability.

Summary updated

BibTeX
@inproceedings{aljabari2024event,
  author = {Alaa Aljabari and Lina Duaibes and Mustafa Jarrar and Mohammed Khalilia},
  title = {{Event-Arguments Extraction Corpus and Modeling using BERT for Arabic}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}

Loading related content...

Figure