TL;DR
This paper introduces "narrative chain embeddings" as a novel way to model stories, moving beyond just understanding words to understanding narrative structures. They propose a new dataset of German narrative chains and demonstrate state-of-the-art performance on the narrative cloze task, showing their embeddings can capture narrative similarity better than traditional methods.
What Problem Does This Paper Address?
Understanding and computationally modeling narratives is a complex challenge. While language models excel at understanding sentence-level semantics, capturing the overarching structure and flow of a story remains difficult. Traditional approaches often focus on inferring explicit narrative schemas (like Propp's model) or rely on masked language modeling, which primarily focuses on word prediction. This paper aims to bridge this gap by using the "narrative cloze task" – predicting missing events in a narrative chain – as a training objective to learn narrative understanding rather than just language understanding. This could enable computational comparisons of stories, a task with potential applications in digital humanities and beyond.
Methods
The core of the paper lies in their approach to learning "narrative chain embeddings." They propose two main methods:
- Static Embedding Approach: This method uses a transformer architecture with static word embeddings. Crucially, it takes a twofold approach to entity representation, combining word embeddings of entity surface forms with one-hot encodings. The training objective is inspired by masked language modeling, where the model predicts a missing verb lemma. A key innovation here is the use of a cosine-similarity-based loss function, which they found to be more effective than a standard cross-entropy loss for learning from ambiguous training examples.
- Language Model Approach: For comparison, they also employ a state-of-the-art language model, T5. They convert narrative chains into textual representations and use T5's denoising objective. Embeddings are derived from the last encoder state of the T5 model.
Both approaches are trained on narrative chains, which are sequences of triples (subject, verb, object) that share a common participant (protagonist).
Datasets
The authors utilize and introduce several datasets:
- Gigaword Dataset: This dataset, preprocessed by Granroth-Wilding and Clark (2016), serves as a source for English narrative chains.
- New German Dataset: A novel dataset of German narrative chains is introduced, extracted from German news data using a coreference resolution system and dependency parsing. This dataset contains approximately 1.8 million chains.
- SemEval Dataset (Chen et al., 2022a): This multilingual news similarity dataset is used for extrinsic evaluation. It contains human annotations of article similarity across seven dimensions, including narrative schema similarity.
Key Insights and Ideas
- Narrative Cloze as a Training Objective: The paper argues that using the narrative cloze task as a training objective can lead to embeddings that capture narrative understanding, distinct from general language understanding.
- Cosine Similarity Loss for Embeddings: The cosine-similarity-based loss function proved significantly more effective than cross-entropy for learning from ambiguous narrative data, suggesting it better handles semantically similar verbs.
- Entity Information's Impact: While removing entity surface forms slightly degrades performance for their static embedding model, it drastically impacts the T5 model, suggesting T5 might rely more on surface-level matching rather than deeper narrative understanding. This highlights the potential for their approach to focus more on narrative structure over specific entities.
- Limitations of Current Evaluation: The SemEval dataset, while useful, shows a high correlation between narrative similarity and general text similarity. This makes it difficult to definitively prove the need for specialized narrative modeling, suggesting a need for datasets in domains where this correlation is weaker.
- Sequential Information Matters (to an extent): Shuffling the order of triples in a chain had a minor impact on performance, indicating that while narrative cloze models don't strictly rely on sequential order, some sequential information is captured and beneficial.
Key Results
- State-of-the-Art MCNC Performance: Their models achieve state-of-the-art results on the Multiple Choice Narrative Cloze (MCNC) task for both English and German datasets, outperforming previous approaches by significant margins (e.g., over 1.8 percentage points on their German dataset).
- Extrinsic Evaluation Challenges: Despite strong MCNC performance, their embeddings perform "rather poorly" in identifying narratively similar texts on the SemEval dataset. This suggests a potential disconnect between the narrative cloze task and true narrative similarity judgment.
- T5's Overperformance with Surface Forms: The T5 model achieves very high accuracy (over 92%) when entity surface forms are included, but this performance drops dramatically (over 60 percentage points) when they are removed. This suggests T5 might be leveraging entity matching rather than narrative structure.
- German Model Performance: The German model generally exhibits better performance, potentially due to improvements in the extraction pipeline.
Limitations and Future Work
- Limited Evaluation Data: The authors acknowledge that the SemEval dataset's high correlation between narrative and general similarity makes it challenging to demonstrate the unique value of narrative modeling. They plan to address this by seeking datasets in other domains.
- Information Scarcity in Chains: A significant limitation identified is that narrative chains, as currently extracted, may provide too little information for robust semantic modeling.
- Narrative Cloze Task Suitability: The paper questions whether the narrative cloze task, in its current form, is the ideal objective for building semantic representations of narratives.
- Future Directions:
- Investigating contextual embeddings (like Wilner et al., 2021) and explicitly removing actor information.
- Exploring alternative ways to model narratives beyond the current cloze task.
- Developing improved event representations.
- Potentially leveraging pre-trained large language models.
- Pursuing supervised narrative similarity data, while being mindful of the challenge posed by common entity names.
Summary updated