Thinking about how to extract: Energizing LLMs' emergence capabilities for document-level event argument extraction

Kai Shuang, Zhouji Zhouji, Qiwei Wang, Jinyu GuoACL 2024 · OpenReview

Abstract

Document-level event argument extraction (D-EAE) faces two primary challenges: key feature forgetting in long contexts and cross-event argument confusion. This paper introduces EAESR (Event Argument Extraction based on guided Summarization and Reasoning), a framework that leverages the emergent capabilities of Large Language Models (LLMs) to address these issues. By generating guided summaries and inter-event reasoning, EAESR highlights critical event information and clarifies complex associations. Experimental results on WIKIEVENTS and RAMS benchmarks show that EAESR achieves new state-of-the-art performance, particularly in few-shot settings where it outperforms baselines by an average of 11% F1.

Introduction

Event Argument Extraction (EAE) aims to identify the specific roles (the "who, what, where, and when") associated with event triggers. Moving from sentence-level to document-level extraction introduces significant complexity. First, key feature forgetting occurs when relevant arguments are buried within long, noisy documents. Second, cross-event argument confusion arises when multiple events share arguments or when arguments are scattered across different sentences, requiring sophisticated reasoning to link them correctly.

Existing approaches, such as sliding windows or graph neural networks, often fail to capture the global context or the nuanced logic needed for implicit links. EAESR addresses these by using LLMs not as direct extractors, which are prone to hallucination, but as sophisticated feature generators that provide a "logical map" for the extraction process.

Methodology

EAESR integrates LLM-driven insights into a traditional extraction pipeline through three core components:

  1. Guided Summarization: The LLM produces a condensed version of the document tailored to specific event types. This summary is transformed into an Abstract Meaning Representation (AMR) graph, which a Graph Convolutional Network (GCN) then processes to capture global semantic features.
  2. Reasoning Feature Extraction: Utilizing Chain-of-Thought (CoT) prompting, the LLM explains the logic behind identifying specific roles. This provides the model with explicit inter-event associations.
  3. Entity Information: The LLM identifies potential argument boundaries, enhancing the model's sensitivity to argument spans.
The EAESR framework architecture, integrating LLM-driven summarization and reasoning features into an attention fusion layer.

These features are combined with BERT-based sentence encodings in an Attention Fusion Layer to make the final argument predictions.

Experimental Results

EAESR was evaluated on the RAMS and WIKIEVENTS datasets, achieving a new state-of-the-art (SOTA) on both. It improved Argument Classification F1 (Arg-C) by 1.6% on RAMS and 1.3% on WIKIEVENTS over previous best-performing models.

Few-Shot Performance

The most significant advantage of EAESR is observed in low-data scenarios. The reasoning features generated by the LLM allow the model to learn the underlying logic of extraction rather than just surface patterns. As shown in the performance analysis, EAESR trained on only 10 random samples can outperform competitive baselines like APE trained on 200 samples.

Few-shot performance comparison: EAESR maintains high F1 scores even with extremely limited event records compared to baseline models.

Ablation studies confirmed that the reasoning component is the most critical; removing it led to the steepest decline in performance, highlighting the value of "thinking step-by-step" guidance.

Conclusion

EAESR demonstrates that LLMs can be effectively used as reasoning assistants to overcome the inherent noise and complexity of document-level extraction. While the current framework faces challenges regarding inference latency and potential error propagation from LLM summaries, it sets a new benchmark for D-EAE. Future work will focus on utilizing more lightweight models and developing joint extraction methods to further refine global information capture.

Summary updated

Abstract

There are two key challenges remaining for the document-level event argument extraction (D-EAE) tasks: key feature forgetting and cross-event argument confusion. The emergence capability of large language models (LLMs) holds promise for solving the above two challenges. In this paper, we propose a document-level event argument extraction method based on guided summarization and reasoning (EAESR), which leverages the emergence capabilities of LLMs to highlight key event information and to clarify the explicit and implicit association between multiple events. Specifically, we generate document summarization information that shorten the length of the event context while preserving the key event features. In addition, we generate inter-event reasoning information, which helps EAESR make sense of the correlations between events and reduces their dependence on the event context, especially to better cope with the few-shot D-EAE task. Then, we obtain named entity information to enable EAESR to learn argument boundary features to improve the sensitivity of its argument boundary recognition. Eventually, we fused the above features and sentence features to make EAESR have summarizing and reasoning capabilities simultaneously. Extensive experiments on WIKIEVENTS and RAMS have shown that EAESR achieves a new state-of-the-art that outperforms the baseline models by 1.3% F1 and 1.6% F1, respectively, and averages 11% F1 in few-shot settings.
BibTeX
@inproceedings{shuang2024thinking,
  author = {Kai Shuang and Zhouji Zhouji and Qiwei Wang and Jinyu Guo},
  title = {{Thinking about how to extract: Energizing LLMs' emergence capabilities for document-level event argument extraction}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=cDrGqVnWY5},
}
Figure