REGen: A Reliable Evaluation Framework for Generative Event Argument Extraction

Omar Sharif, Joseph Gatto, Madhusudan Basak, Sarah Masud PreumEMNLP 2025

TL;DR

Current methods for evaluating event argument extraction (EAE) using exact matching severely underestimate the performance of modern generative models like LLMs. This paper introduces REGen, a novel evaluation framework that combines exact, relaxed, and LLM-based matching to provide a more reliable and human-aligned assessment, demonstrating significant performance gains and efficiency improvements.

What Problem Does This Paper Address?

Event argument extraction (EAE) is a crucial task in natural language processing, aiming to identify and extract specific pieces of information (arguments) related to events described in text. For example, in the sentence "Alice sent a package to Bob on Monday," the event is "Send," and arguments include "Alice" (sender), "Bob" (recipient), and "Monday" (time).

Traditionally, EAE evaluation relies on Exact Match (EM), where a predicted argument must perfectly match the ground-truth annotation, character for character. While this works well for simpler, extractive models, it falls short for generative models, especially Large Language Models (LLMs). LLMs often produce semantically correct arguments that might differ slightly in wording, structure, or even be composed of information scattered across a document. EM's strictness leads to a severe underestimation of these models' true capabilities, failing to capture valid variations, implicit arguments, or arguments spread across multiple sentences. This gap necessitates a more robust evaluation framework.

Methods

REGen proposes a multi-stage evaluation framework designed to be more aligned with human judgment while minimizing computational costs:

  1. Exact Match (EM): This initial stage filters out arguments that match the ground truth perfectly. This is the most reliable and requires no human evaluation, efficiently handling obvious matches.
  2. Relaxed Match (RM): Arguments not matched in EM are then evaluated using embedding-based similarity. If the semantic similarity between a predicted and ground-truth argument exceeds a predefined threshold (0.85 in this paper, determined through experimentation), it's considered a relaxed match. This stage accounts for minor syntactic variations.
  3. Complex Match (CM): Arguments still unmatched after RM are passed to this stage. Here, a pre-selected LLM (GPT-40 in their experiments) acts as a judge to determine semantic similarity based on context, even with significant lexical or syntactic differences. This stage requires human evaluation to validate the LLM judge's performance.
  4. Alignment with Human Judgments (JAM Score): To account for potential misjudgments in RM and CM, REGen introduces a Judgment Aligned Match (JAM) score. This score systematically penalizes counts based on the deviation rate between the LLM judge's predictions and human annotations at each level. This ensures the final score is a more reliable reflection of true performance, acting as a conservative estimate.

Datasets

The authors evaluated REGen on six widely-used EAE datasets:

  • DiscourseEE: Online health discussions, including explicit, implicit, and scattered arguments.
  • PHEE: Pharmacovigilance domain.
  • RAMS: News domain.
  • GENEVA: General domain, covering books, articles, journals, and Wikipedia.
  • DocEE: Long news documents.
  • WikiEvents: Wikipedia texts, known for being argument-dense.

They used the complete official test sets for these datasets, unlike prior work that often sampled subsets, providing a more comprehensive evaluation.

Key Insights and Ideas

  • Exact Match is Insufficient for LLMs: The core insight is that traditional EM evaluation is fundamentally misaligned with the generative capabilities of LLMs, leading to misleadingly low performance scores.
  • LLMs as Judges are Promising but Need Validation: While LLMs can effectively judge argument similarity, their judgments are not perfect and require alignment with human evaluations to ensure reliability.
  • A Hierarchical Approach is Key: REGen's strength lies in its layered approach, starting with the most reliable (EM) and progressively incorporating less reliable but more comprehensive methods (RM, CM), all while being anchored by human judgment.
  • Context is Crucial: Many argument variations that EM misses are semantically correct when considered within the document's context, highlighting the need for context-aware evaluation.

Key Results

  • Significant Performance Gains: REGen revealed an average F1 score improvement of +23.93 points across all datasets compared to EM.
  • High Human Alignment: The REGen framework achieved 87.67% alignment with human assessments of argument correctness.
  • Efficiency Improvement: REGen reduced inference costs by 41.2% compared to a pure LLM-as-judge approach, making evaluation more practical.
  • Reliability: The JAM score ensures that the reported performance is more reliable and less prone to overestimation due to lenient matching criteria.

Limitations and Future Work

  • No Statistical Significance Testing: The authors did not conduct statistical significance testing due to time and computational cost constraints, focusing instead on demonstrating performance gaps.
  • No Prompt Optimization: For fairness and practicality, a single prompt was used across all models and datasets. Optimizing prompts for each model and dataset could further improve performance.
  • Future Work: The authors suggest exploring few-shot experiments, finding optimal prompting strategies, and investigating smaller fine-tuned judge models. They also believe REGen can pave the way for evaluating other generative IE tasks like relation and entity extraction.

Summary updated

BibTeX
@inproceedings{sharif2025regen,
  author = {Omar Sharif and Joseph Gatto and Madhusudan Basak and Sarah Masud Preum},
  title = {{REGen: A Reliable Evaluation Framework for Generative Event Argument Extraction}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure