EventRelBench: A Comprehensive Benchmark for Evaluating Event Relation Understanding in Large Language Models

Jie Gong, Biaoshuai Zheng, Qiwang HuEMNLP 2025

TL;DR

This paper introduces EventRelBench, a new benchmark designed to rigorously evaluate how well Large Language Models (LLMs) understand and extract complex relationships between events in text. They found that even state-of-the-art LLMs struggle with this task, prompting them to create EventRelInst, a fine-tuning dataset that significantly boosts LLM performance on event relation extraction.

What Problem Does This Paper Address?

Understanding how events relate to each other (e.g., one event causing another, or two events happening at the same time) is fundamental for many Natural Language Processing (NLP) tasks like narrative comprehension and information extraction. While LLMs have shown impressive capabilities across various NLP tasks, there's been a lack of systematic evaluation for their ability to grasp these intricate event relationships. Existing benchmarks often focus on single relation types or sentence-level analysis, leaving a gap in understanding LLMs' performance across diverse event relation categories and at document-level scope. This makes it hard to pinpoint specific weaknesses and develop targeted improvements.

Methods

The authors tackle this problem in two main ways:

  1. EventRelBench Benchmark: They constructed a comprehensive benchmark comprising 35,000 multiple-choice questions covering four key event relation types:
  2. Coreference: When different phrases refer to the same event.
  3. Temporal: The time order of events (before, after, equal, vague).
  4. Causal: When one event causes or is caused by another.
  5. Supersub: Hierarchical relationships where one event is a component of another. The benchmark includes questions at both sentence and document levels, offering a multi-faceted evaluation.

  6. EventRelInst Dataset: To address the observed limitations of LLMs, they created a 48,000-example instruction fine-tuning dataset. This dataset reformats domain-specific event relation data into explicit instruction-response pairs, aiming to help models better grasp the nuances of each relation type and its contextual granularity.

They evaluated various LLMs using four prompting strategies: zero-shot, zero-shot with Chain-of-Thought (CoT), few-shot, and few-shot with CoT.

Datasets

  • EventRelBench: A newly introduced benchmark with 35,000 multiple-choice questions covering four event relation types (coreference, temporal, causal, supersub) at sentence and document levels. It draws from existing datasets like ECB+, EventStoryLine, MATRES, TCR, Causal-TimeBank, MAVEN-ERE, and HiEve.
  • EventRelInst: A newly introduced instruction fine-tuning dataset with 48,000 examples, specifically designed to improve LLM performance on event relation extraction.
  • Existing Datasets (used for EventRelBench construction): ECB+, EventStoryLine, MATRES, TCR, Causal-TimeBank, MAVEN-ERE, HiEve.

Key Insights and Ideas

  • LLMs Still Struggle: Despite advancements, current general-purpose LLMs show significant limitations in accurately extracting and understanding event relationships, with some even underperforming random guessing.
  • Document-Level Advantage: LLMs perform better on document-level tasks than sentence-level tasks, suggesting their strength in capturing long-range contextual information.
  • Fine-tuning is Key: Instruction fine-tuning with domain-specific data (like EventRelInst) provides substantial performance gains, even for smaller models.
  • Prompting Strategies Matter, But with Caveats: While few-shot prompting can help, CoT prompting doesn't always improve performance and can sometimes introduce noise.
  • Model Size Isn't Everything: Architecture, training data, and fine-tuning strategy play a more critical role in performance than sheer model size.

Key Results

  • Overall Performance: The fine-tuned model Qwen2.5-7B-FT achieved the highest overall accuracy (60.0%) and F1 score (54.5%) on EventRelBench, significantly outperforming its base version.
  • Base Model Performance: Among models without fine-tuning, DeepSeek-V3 and ERNIE-3.5-8K showed strong out-of-the-box performance, with DeepSeek-V3 demonstrating remarkable consistency across prompting styles.
  • Underperformers: Models like AquilaChat-7B and Gemma-7B-IT consistently underperformed, highlighting the need for specialized tuning.
  • Fine-tuning Gains: Fine-tuning with EventRelInst led to significant improvements, with Qwen2.5-7B-FT showing a +6.9 accuracy and +10.4 F1 improvement over its base model.
  • Relation-Specific Performance:
    • Causal Relation: Fine-tuned models showed a clear edge, with Qwen2.5-7B-FT reaching over 73% accuracy/F1.
  • Coreference Relation: LLMs demonstrated robust capability, with several untuned models already surpassing 65% accuracy. Fine-tuning further boosted performance.
  • Supersub Relation: This was found to be more challenging, with fine-tuning yielding only modest improvements.
  • Temporal Relation: This proved to be the hardest relation type, with most LLMs achieving F1 scores below 30%, even after fine-tuning.

Limitations and Future Work

  • Limited Scope: EventRelBench covers four fundamental relation types but doesn't capture the full diversity and granularity of event interactions.
  • English-Centric: The benchmark and dataset are primarily constructed from English text, and their applicability to non-English languages is untested.
  • Annotation Nuances: Subtle ambiguities in defining event boundaries and relation strengths might have introduced inconsistencies.
  • Potential Overfitting: Fine-tuning might lead to overfitting to specific prompt formulations, potentially hindering zero-shot adaptation to novel styles.
  • Model Space: The evaluation covered a representative spectrum of LLMs but not the entire range of model scales, architectures, or inference setups.

Future Work Suggestions:

  • Enhance prompt design for temporal relations with timestamp normalization, interval calculation, and narrative chain tasks.
  • Incorporate recursive and cross-domain hierarchy instances for supersub relation extraction.
  • Expand to non-English languages.
  • Explore a wider range of LLMs and inference strategies.
  • Address potential overfitting by ensuring diverse training data.

Summary updated

BibTeX
@inproceedings{gong2025eventrelbench,
  author = {Jie Gong and Biaoshuai Zheng and Qiwang Hu},
  title = {{EventRelBench: A Comprehensive Benchmark for Evaluating Event Relation Understanding in Large Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure