Abstract
Traditional event extraction (EE) relies heavily on large-scale human-annotated datasets, making it difficult to scale to new event types. This paper investigates zero-shot event extraction by repurposing pretrained models for Textual Entailment (TE) and Question Answering (QA). While this approach outperforms prior zero-shot benchmarks, a significant performance gap remains compared to supervised models. We provide a detailed error taxonomy to identify the primary bottlenecks in current "off-the-shelf" NLP models for event structure extraction.
Introduction
Event Extraction (EE) is the task of identifying "what happened" in a text, the participants involved, and their specific roles. For instance, in the sentence "China purchased submarines from Russia," an EE system must identify the trigger ("purchased"), the event type (Transfer-Ownership), and the arguments ("China" as Buyer, "Russia" as Seller).
The primary challenge in EE is the high cost of annotation. To address this, we explore a zero-shot setting where no event-specific training data is used. Instead, we leverage the general linguistic knowledge captured by models trained on broader tasks like Textual Entailment (TE) and Question Answering (QA).
Methodology
We propose a two-stage pipeline that treats event extraction as a series of logical tests and questions:
- Trigger Extraction (T-Ext): Candidate verbs and nouns identified via Semantic Role Labeling (SRL) are evaluated using a TE model. The sentence serves as the "premise," and the system generates a "hypothesis" (e.g., "This text is about a transfer of ownership"). High entailment probability indicates a valid trigger.
- Argument Extraction (A-Ext): For identified triggers, an extractive QA model asks predefined questions based on the event template (e.g., "Who bought something?"). The returned spans are treated as event arguments.
Experimental Setup
Models were pretrained on standard benchmarks: MNLI (for TE), BoolQ (for Yes/No QA), and QAMR/SQuAD2.0 (for extractive QA). Evaluation was conducted on the ACE-2005 and ERE datasets, the gold standards for event extraction.
Analysis and Results
Our system achieved an F1 score of 16.8 on ACE-2005 in a full zero-shot setting, outperforming previous zero-shot attempts. However, a massive gap remains compared to supervised state-of-the-art models (~56.8 F1).
Error Taxonomy
To understand this gap, we categorized errors into four main types:
- Model Weakness (Model): Models struggle with fine-grained semantic distinctions (e.g., "Attack" vs. "Injure").
- Usage Error (Usage): Distracting context can mislead models (e.g., unrelated keywords triggering false positives).
- Task/Annotation Ambiguity (Task): Inconsistencies in gold standard annotations penalize zero-shot models.
- The "SQuAD" Trap: Models trained on long-form context (like SQuAD) often perform worse on sentence-level extraction than those trained on single-sentence tasks (like QAMR).
Conclusion and Future Work
Trigger identification remains the primary bottleneck in zero-shot EE. Future research should focus on improving trigger span identification and developing better "minimal-pair" reasoning to enhance context sensitivity. Additionally, while contextualized questions improve performance, they must be carefully designed to avoid ungrammaticality that confuses QA models. Bridging the gap between broad logical reasoning and rigid event ontologies remains a key challenge for the field.
Summary updated