ESTER: A Machine Reading Comprehension Dataset for Event Semantic Relation Reasoning

Rujun Han, I-Hung Hsu, Jiao Sun, Julia Baylon, Qiang Ning, Dan Roth, Nanyun Peng

Introduction

While Natural Language Processing (NLP) has achieved significant success in identifying entities and temporal sequences, true narrative intelligence requires a deeper understanding of how events are semantically linked. Existing Event Relation Extraction (RE) tasks often rely on rigid classification labels that fail to capture the nuances of human reasoning. To address this, we introduce ESTER, a Machine Reading Comprehension (MRC) dataset that frames event reasoning as a natural language question-answering task.

The ESTER Dataset

ESTER focuses on five key event semantic relations: Causal, Conditional, Counterfactual, Sub-event, and Coreference. Unlike traditional datasets, ESTER uses a Question Answering (QA) format, which is more aligned with how humans naturally reason about narratives.

Example of ESTER's QA format across different event relation types, showing the source passage and corresponding questions.

The dataset is built upon news articles from the TempEval3 (TE3) workshop and contains over 6,000 questions capturing 10.1K event relation pairs. This scale and variety provide a robust benchmark for testing a model's ability to navigate complex event hierarchies and motivations.

Methodology

The researchers formulated event reasoning as a Generative QA task. In this setup, a model is provided with a passage and a natural language query (e.g., "What caused the meeting?"). The model must then generate the correct event span from the text. This approach was compared against traditional Extractive QA (e.g., RoBERTa-large), which predicts start and end tokens for answer spans.

To ensure high-quality annotations, a rigorous five-stage worker validation and training process was implemented. This ensured that human annotators could accurately distinguish between subtle relations like "Causal" versus "Conditional," which is critical for a high-quality reasoning benchmark.

Key Results and Insights

The benchmark results indicate that ESTER is a significant "stress test" for modern NLP models. The best-performing model, UnifiedQA-large, achieved an F1 score of 63.3% and a HIT@1 of 83.5%. However, a substantial gap remains when compared to human performance (79.6% F1 and 100% HIT@1).

One critical finding is that generative models consistently outperform extractive ones. Extractive models often produce fragmented or semantically incomplete spans, whereas generative models yield more coherent event descriptions.

Confusion matrix showing model performance across different event relation types, highlighting the difficulty of Sub-event and Coreference relations.

Analysis of specific relation types reveals that models struggle most with Sub-event and Coreference relations. As shown in the confusion matrix, while models are relatively proficient at identifying Causal and Conditional links, hierarchical and coreferential reasoning—which require understanding shared participants and event composition—remain challenging.

Conclusion

ESTER highlights a clear "reasoning gap" in current AI models. The plateauing of performance after 2,000 training examples suggests that simply increasing data volume is insufficient. Future research must focus on developing architectures capable of more sophisticated event-centric reasoning and exploring ways to transfer knowledge from related tasks like temporal ordering to improve performance on semantic relations.

Summary updated

BibTeX
@inproceedings{han2021ester,
  author = {Rujun Han and I-Hung Hsu and Jiao Sun and Julia Baylon and Qiang Ning and Dan Roth and Nanyun Peng},
  title = {{ESTER: A Machine Reading Comprehension Dataset for Event Semantic Relation Reasoning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.597},
}
Figure