TL;DR
This paper introduces a novel task: automatically identifying "new" events in a narrative as they unfold. They define "new" events as those that provide novel information not previously mentioned or inferable through common sense. The authors created a new annotated dataset and developed baseline models to tackle this challenging problem, which is crucial for deeper narrative understanding and downstream AI applications.
What Problem Does This Paper Address?
Understanding narratives is a complex task for AI, and a key aspect is grasping the sequence of events. While previous work has focused on extracting events or linking them, this paper tackles a more nuanced problem: distinguishing truly new information from repeated or inferable events within a narrative. Imagine reading a story; you want to know when something genuinely surprising or informative happens, not just a rehash of what you already know. This "Information Status" of events is crucial for summarizing, reasoning about, and engaging in dialogue about narratives. The authors aim to equip AI with this ability to discern what's genuinely new in a story.
Methods
The core of their approach involves defining and annotating "new" events. An event is defined as a triplet of (subject, predicate, object). They then developed a process to identify these new events:
- Event Candidate Extraction: They used an unsupervised model to extract potential event candidates (subject-predicate-object triplets) from narratives.
- Annotation Task: Human annotators were presented with narratives sentence by sentence, along with candidate events. They had to determine if a candidate event was "discourse-new" (not previously mentioned) and not inferable through common sense. Annotators could also add new events as free-form text if the model missed them.
- Baseline Models: To evaluate their annotation and the task's difficulty, they developed both rule-based and neural network baselines. These baselines were tested in two settings:
- Candidate Selection: Models select "new" events from the extracted candidates.
- Sequence Tagging: Models tag tokens within a sentence that represent new events.
Datasets
The primary dataset used is the SEND dataset (Stanford Emotional Narratives Dataset). This is a publicly available corpus of emotional narratives collected from individuals recounting their positive and negative experiences. The authors created a new annotated version of this dataset specifically for the "new event detection" task. They provide this annotated dataset, along with their annotation materials and baseline models, to the research community.
Key Insights and Ideas
- "Newness" is Multifaceted: The paper highlights that an event is "new" not just if it's mentioned for the first time, but also if its information cannot be inferred through common sense. This dual criterion is key to capturing genuine novelty.
- Human Annotation is Crucial: The task is complex enough that human annotators are essential for creating a high-quality dataset. The authors' detailed annotation protocol and analysis of inter-annotator agreement demonstrate the challenges and nuances involved.
- Model Limitations: Even sophisticated models struggle to capture all new events. The authors found that their baseline model often missed events that annotators identified and had to add as "continuous spans," underscoring the difficulty of event extraction itself.
- Narrative Structure Matters: The analysis of where new events tend to appear (e.g., more in the second half of sentences) suggests that narrative structure can provide clues for identifying novelty.
Key Results
- The annotation process yielded a dataset where a significant portion of new events were identified as "continuous spans" by annotators, indicating that the initial candidate extraction model was not exhaustive.
- Among the rule-based baselines, "Last Candidate Selector" performed best in precision for candidate selection, while "New Entity Selector" achieved the highest recall but with low precision.
- Neural models (BERT and RoBERTa) significantly outperformed rule-based baselines in both candidate selection and sequence tagging tasks. RoBERTa achieved the best F1 score (54.3%) in candidate selection and (67.1%) in sequence tagging (when trained on both candidates and spans).
- The sequence tagging models showed higher recall than precision, suggesting they are good at identifying potential new events but sometimes over-tagging.
Limitations and Future Work
- Dataset Specificity: The current dataset consists of personal narratives in English, collected in a controlled "in-vitro" setting. Future work should extend this to other languages, genres, and more naturalistic conversational data.
- Annotation Variability: The annotation was done by five internal annotators, not crowd-sourced. This might introduce some variability, and future work could explore how crowd-sourcing affects annotation quality and agreement.
- IOB Tagging: The sequence tagging task was simplified to a binary tagging task. Developing models for more complex IOB tagging of overlapping spans is suggested as future work.
- Improving Precision: For sequence tagging, the authors note that while recall is high, precision could be improved, suggesting further research into features that better distinguish true new events.
Summary updated