Introduction
While Natural Language Processing (NLP) has achieved significant success in identifying entities and temporal sequences, true narrative intelligence requires a deeper understanding of how events are semantically linked. Existing Event Relation Extraction (RE) tasks often rely on rigid classification labels that fail to capture the nuances of human reasoning. To address this, we introduce ESTER, a Machine Reading Comprehension (MRC) dataset that frames event reasoning as a natural language question-answering task.
The ESTER Dataset
ESTER focuses on five key event semantic relations: Causal, Conditional, Counterfactual, Sub-event, and Coreference. Unlike traditional datasets, ESTER uses a Question Answering (QA) format, which is more aligned with how humans naturally reason about narratives.
The dataset is built upon news articles from the TempEval3 (TE3) workshop and contains over 6,000 questions capturing 10.1K event relation pairs. This scale and variety provide a robust benchmark for testing a model's ability to navigate complex event hierarchies and motivations.
Methodology
The researchers formulated event reasoning as a Generative QA task. In this setup, a model is provided with a passage and a natural language query (e.g., "What caused the meeting?"). The model must then generate the correct event span from the text. This approach was compared against traditional Extractive QA (e.g., RoBERTa-large), which predicts start and end tokens for answer spans.
To ensure high-quality annotations, a rigorous five-stage worker validation and training process was implemented. This ensured that human annotators could accurately distinguish between subtle relations like "Causal" versus "Conditional," which is critical for a high-quality reasoning benchmark.
Key Results and Insights
The benchmark results indicate that ESTER is a significant "stress test" for modern NLP models. The best-performing model, UnifiedQA-large, achieved an F1 score of 63.3% and a HIT@1 of 83.5%. However, a substantial gap remains when compared to human performance (79.6% F1 and 100% HIT@1).
One critical finding is that generative models consistently outperform extractive ones. Extractive models often produce fragmented or semantically incomplete spans, whereas generative models yield more coherent event descriptions.
Analysis of specific relation types reveals that models struggle most with Sub-event and Coreference relations. As shown in the confusion matrix, while models are relatively proficient at identifying Causal and Conditional links, hierarchical and coreferential reasoning—which require understanding shared participants and event composition—remain challenging.
Conclusion
ESTER highlights a clear "reasoning gap" in current AI models. The plateauing of performance after 2,000 training examples suggests that simply increasing data volume is insufficient. Future research must focus on developing architectures capable of more sophisticated event-centric reasoning and exploring ways to transfer knowledge from related tasks like temporal ordering to improve performance on semantic relations.
Summary updated