Language Model Priming for Cross-Lingual Event Extraction

Steven Fincke, Shantanu Agarwal, Scott Miller, Elizabeth BoscheeAAAI 2022 · DOI

Introduction

Event Extraction (EE) involves identifying specific occurrences (triggers) and their associated participants (arguments) within unstructured text. While performance on high-resource languages like English has improved significantly, cross-lingual transfer—extracting events from languages with limited annotated data—remains a major challenge. Standard contextualized language models like BERT often produce static representations for tokens, regardless of the specific extraction task. This paper introduces IE-PRIME, a method that "primes" the model by prepending task-specific queries to the input, enabling dynamic, task-aware representations.

Methodology: The IE-PRIME Approach

IE-PRIME builds upon a standard architecture consisting of a Transformer, a bi-LSTM, and a CRF layer. The core innovation lies in the input formatting, which forces the model to attend to the sentence through the lens of a specific trigger or role.

Trigger Priming

To identify if a word acts as an event trigger, the candidate word is prepended to the sentence. This "primes" the Transformer's self-attention mechanism to focus on the context relevant to that specific word's potential as a trigger.

Trigger priming prepends the candidate word to the input sentence to focus the model's attention.

Argument Priming

For argument extraction, the model prepends both the trigger and the specific argument role being queried. To maintain language agnosticism and avoid English-centric bias during cross-lingual transfer, roles are represented as unique integer strings (e.g., "13" for "Defendant") rather than natural language labels.

Argument priming uses the trigger and a language-agnostic integer ID for the target role.

By using BIO (Beginning, Inside, Outside) tagging at the output, the model can identify multiple argument spans for a single role simultaneously, providing more flexibility than standard question-answering heads.

Experimental Setup and Results

The researchers evaluated IE-PRIME using the ACE 2005 corpus, focusing on English-to-Arabic zero-shot transfer. In this setting, the model is trained exclusively on English data and tested on Arabic without any exposure to Arabic event annotations.

The results demonstrate a significant leap in performance:

  • Cross-Lingual Argument Classification: IE-PRIME achieved an F1 score of 50.3, substantially outperforming previous state-of-the-art models (which ranged from 36.4 to 44.5).
  • Trigger Detection: The priming technique improved cross-lingual trigger identification F1 from 42.4 to 51.0.
  • Low-Resource Efficiency: The method is particularly robust when training data is scarce. IE-PRIME trained on only 20% of English data performed nearly as well as a baseline model trained on the full dataset.

Discussion and Key Insights

The success of IE-PRIME highlights the importance of representational flexibility in structured prediction tasks. By prepending the query directly to the input, the language model can generate distinct vector representations for the same sentence tokens depending on the specific "question" being asked (e.g., is this token a victim, a perpetrator, or a location?). This approach is essentially a form of "soft prompting" that allows the Transformer's self-attention mechanism to react more effectively to the change in focus.

Furthermore, the use of integer IDs for roles is a critical design choice for cross-lingual robustness. By avoiding natural language labels like "Defendant," the model is prevented from developing an English-centric semantic bias. Instead, it is forced to learn the underlying structural relationships of events, which are more likely to be consistent across different languages. This language-agnostic approach is a key driver behind the model's superior performance in zero-shot transfer scenarios.

Limitations and Future Directions

Despite its strengths, IE-PRIME currently operates at the sentence level, potentially missing document-wide context that could clarify ambiguous roles. Additionally, while effective, the current pipeline handles triggers and arguments in separate steps. Future work may explore joint modeling to improve efficiency and capture inter-dependencies between triggers and their arguments.

Summary updated

BibTeX
@inproceedings{fincke2022language,
  author = {Steven Fincke and Shantanu Agarwal and Scott Miller and Elizabeth Boschee},
  title = {{Language Model Priming for Cross-Lingual Event Extraction}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i10.21307},
}
Figure