ACE-2005-PT: Corpus for Event Extraction in Portuguese

Luís Filipe Cunha, Purificação Silvano, Ricardo Campos, Alípio JorgearXiv 2024 · DOI · arXiv

TL;DR

This paper introduces ACE-2005-PT, the first comprehensive corpus for event extraction in Portuguese, featuring both European and Brazilian variants. By developing a sophisticated translation and annotation alignment pipeline, the authors successfully ported the gold-standard ACE-2005 dataset to Portuguese, achieving a relaxed alignment match score of 87.77% against human expert verification.

What Problem Does This Paper Address?

Event extraction is a fundamental Natural Language Processing (NLP) task that involves identifying an event's central word (the trigger) and its associated participants or context (the arguments). For instance, in the sentence "Marie Curie was born in Warsaw," "born" is the trigger (type: Life:Be-Born), while "Marie Curie" (Person) and "Warsaw" (Place) are arguments.

The gold standard for benchmarking these systems is the ACE-2005 corpus. However, it is only available in English, Chinese, and Arabic. This language barrier severely limits event extraction research in other languages, such as Portuguese.

Simply running the text through Google Translate or DeepL doesn't solve the problem. Translation alters sentence structures, word orders, and word forms. A direct transfer of the original annotation "spans" (the exact character start and end positions) fails roughly half the time because the annotated target words shift or change completely in context.

Methods

To overcome the alignment challenge, the authors built a sequential, multi-stage translation and annotation alignment pipeline. If a simple substring match fails after machine translation, the pipeline tries progressively advanced techniques:

  1. String Match (SMatch): Looks for an exact direct translation of the annotated term within the translated sentence.
  2. Lemmatization (LemmaMatch): Reduces words to their base forms (e.g., verbs to infinitive) to find matches despite morphological variations.
  3. Multiple Translations (MTrans): Leverages the Microsoft Dictionary Lookup API to generate alternative translations and idiomatic expressions.
  4. Synonym Matching: Swaps words for synonyms (used specifically for single-word triggers).
  5. Transformer-based Word Aligner (WAligner): Uses multilingual BERT (mBERT) embeddings to compute a token-level association matrix between the source and target sentences, identifying the most probable corresponding spans.
  6. Fuzzy String Similarity: Employs Levenshtein distance and Gestalt pattern matching via a sliding window to capture multi-word arguments that underwent minor phrasing changes.

The mBERT-driven word alignment matrix helps visually map how tokens correspond across languages:

Figure 1: Alignment matrix from mBERT.

Datasets

  • ACE-2005 (English): The source corpus used for translation.
  • ACE-2005-PT: The newly introduced corpus, mirroring the original's 16,260 sentences and 14,886 annotations. It includes distinct versions for European Portuguese (translated via DeepL) and Brazilian Portuguese (translated via Google Translate).
  • Gold Standard Evaluation Subset: A selection of 1,310 annotations from the test set, manually aligned by an expert linguist to validate the pipeline's accuracy.

Key Insights and Ideas

  • The Multi-Word Hurdle: Aligning triggers (usually single words) is highly successful, but aligning arguments (multi-word phrases) is significantly harder due to syntactic restructuring during translation.
  • Linguistic Divergence (Null Subjects): English requires explicit subjects ("We have been under fire"), but Portuguese allows implicit subjects ("Temos estado sob fogo"). This causes the pipeline to completely lose arguments like "We" because they disappear into verb conjugations.
  • The Contraction Conundrum: In English, determiners are distinct tokens ("in the Middle East"). In Portuguese, prepositions and determiners frequently contract into a single word ("no Médio Oriente", where no = em + o). Because annotation guidelines require including the determiner ("o") but excluding the preposition ("em"), automated tools struggle intensely with these boundaries.

Key Results

The pipeline's performance was evaluated using two metrics: Exact Match (strict character span agreement) and Relaxed Match (token-level F1-score).

  • Overall Pipeline Accuracy: Achieved an 87.77% Relaxed match and a 70.55% Exact match.
  • Triggers vs. Arguments: Triggers were aligned with high precision (93.11% Relaxed / 91.23% Exact), whereas arguments proved much more elusive (85.16% Relaxed / 60.76% Exact).
  • Baseline Downstream Performance: The authors trained a Portuguese Question Answering model on the new corpus for event extraction, scoring a 64.4% F1-score for trigger extraction and 46.7% for argument extraction. While lower than English baselines, it establishes a vital benchmark for future Portuguese NLP.

Limitations and Future Work

The primary limitation stems from systemic language differences, particularly the preposition-determiner contractions in Portuguese, which accounted for a massive 69.14% of all argument alignment errors. Additionally, character-level fuzzy matching occasionally created false positives on short words due to a lack of semantic understanding (e.g., mistakenly matching "We/Nós" with the Portuguese word "no").

For future work, the authors plan to adapt this pipeline to port other crucial NLP datasets, such as the Entities, Relations, and Events (ERE) corpus, into Portuguese and other under-resourced languages.

Summary updated

Abstract

Event extraction is an NLP task that commonly involves identifying the central word (trigger) for an event and its associated arguments in text. ACE-2005 is widely recognised as the standard corpus in this field. While other corpora, like PropBank, primarily focus on annotating predicate-argument structure, ACE-2005 provides comprehensive information about the overall event structure and semantics. However, its limited language coverage restricts its usability. This paper introduces ACE-2005-PT, a corpus created by translating ACE-2005 into Portuguese, with European and Brazilian variants. To speed up the process of obtaining ACE-2005-PT, we rely on automatic translators. This, however, poses some challenges related to automatically identifying the correct alignments between multi-word annotations in the original text and in the corresponding translated sentence. To achieve this, we developed an alignment pipeline that incorporates several alignment techniques: lemmatization, fuzzy matching, synonym matching, multiple translations and a BERT-based word aligner. To measure the alignment effectiveness, a subset of annotations from the ACE-2005-PT corpus was manually aligned by a linguist expert. This subset was then compared against our pipeline results which achieved exact and relaxed match scores of 70.55\% and 87.55\% respectively. As a result, we successfully generated a Portuguese version of the ACE-2005 corpus, which has been accepted for publication by LDC.
BibTeX
@misc{cunha2024ace,
  author = {Luís Filipe Cunha and Purificação Silvano and Ricardo Campos and Alípio Jorge},
  title = {{ACE-2005-PT: Corpus for Event Extraction in Portuguese}},
  year = {2024},
  eprint = {2408.16928},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.16928},
}
Figure