TL;DR
This paper proposes a novel approach to computationally model narrative suspense by framing it as uncertainty reduction over neural representations of stories. Their hierarchical language model, trained on short stories, achieves near human-level accuracy in predicting human judgments of suspense, outperforming previous methods.
What Problem Does This Paper Address?
Suspense is a key element that makes stories engaging and compelling, yet it remains computationally challenging to understand and model. While literary theory has long discussed suspense, NLP systems have only sporadically incorporated it, often relying on planning-based approaches or limited domain knowledge. This paper aims to bridge this gap by developing a computational model that can predict human judgments of suspense, with the ultimate goal of enhancing narrative understanding and generation in NLP systems. The authors explore two theoretical frameworks for suspense: surprise (a backward-looking measure of unexpectedness) and uncertainty reduction (a forward-looking measure of how the story's continuation is made less uncertain).
Methods
The core of the paper is a hierarchical language model built upon Generative Pre-Training (GPT). This model processes stories sentence by sentence, generating contextualized sentence embeddings and their associated probabilities.
- Hierarchical Architecture: The model uses a GPT-based word encoder (
word_enc) to create word embeddings. These are then fed into a sentence encoder (sent_enc) (an RNN) to produce sentence embeddings. Finally, another RNN (story_enc) builds a story embedding from the sequence of sentence embeddings. This hierarchical structure allows for capturing narrative progression. - Suspense Measures:
- Surprise (Hale): Calculated as the negative log probability of the current sentence embedding, given the preceding context.
- Uncertainty Reduction (Hale & Ely): This is the primary focus. It's measured as the change in entropy of the probability distribution over possible next sentences. The authors adapt both Hale's entropy reduction and Ely's measure of expected difference between current and next states. They explore computing these measures directly over sentence embeddings (Ely) or over their probability distributions (Hale).
- Training: The model is trained using a combination of a discriminative loss (
ldisc) to encourage representations that anticipate future sentences and a generative loss (lgen) to improve sentence generation quality. - Inference: For uncertainty reduction, the model generates or samples potential future sentences and computes the change in uncertainty. They also explore using weighted versions of these measures, incorporating sentiment scores to reflect the idea that emotionally charged events are more suspenseful.
Datasets
- WritingPrompts Corpus: A large corpus of circa 300k short stories from the /r/WritingPrompts subreddit. This corpus was chosen for its naturalness and engaging quality, unlike some other NLP story datasets.
- Annotation: 100 stories from the development and test sets were manually annotated by five master workers on a five-point scale for narrative suspense (Big Decrease, Decrease, Same, Increase, Big Increase). This annotation was done sentence-by-sentence.
- TRIPOD Dataset: A corpus of movie synopses annotated with turning points (TPs). This dataset was used for an out-of-domain evaluation to test if suspense models could predict major narrative events.
Key Insights and Ideas
- Uncertainty Reduction is Key: The most significant insight is that uncertainty reduction over story representations (UEly) is the most effective measure for predicting human suspense judgments, outperforming surprise and probability-only measures. This suggests that suspense is not just about what is unexpected, but how the narrative resolves ambiguity about future outcomes.
- Representations Matter More Than Probabilities: While probability distributions are useful, computing uncertainty reduction directly over sentence embeddings (as in the Ely model) proved more effective than relying solely on probability distributions (as in the Hale model). This implies that the semantic content of the story, captured in its embeddings, is crucial for modeling suspense.
- Sentiment Amplifies Suspense: Incorporating sentiment scores (using VADER) as weights in the suspense measures generally improved performance, suggesting that emotionally charged events contribute significantly to the feeling of suspense.
- Hierarchical Models Excel: The hierarchical architecture, which builds representations from words to sentences to the entire story, is crucial for capturing the temporal and contextual nature of narrative.
Key Results
- Near Human-Level Accuracy: The best model, UEly-Gen, achieved a Spearman's $\rho$ of .698 and Kendall's $\tau$ of .610 on the development set, numerically outperforming the human upper bound. On the test set, the correlation was statistically indistinguishable from human performance.
- Outperforming Baselines: Simple baselines like WordOverlap, GloveSim, and GPTSim performed poorly, highlighting the need for more sophisticated narrative representations.
- Turning Point Prediction: The model also showed promise in predicting turning points in movie synopses, demonstrating its ability to identify significant narrative shifts even in an out-of-domain task. The UEly-Cor model achieved a normalized distance of 7.78 on the test set, competitive with existing methods.
Limitations and Future Work
- Limited Event Structure Understanding: The authors acknowledge that their model lacks a higher-level understanding of event structure and primarily relies on textual cues.
- Textual Cues vs. True Understanding: The model might be picking up on superficial textual patterns rather than a deep understanding of narrative causality.
- Future Directions:
- Adversarial Manipulation: Suggests manipulating text (e.g., switching sentence order, paraphrasing) to create adversarial examples for deeper analysis.
- More Sophisticated Annotations: Proposes collecting annotations on reader certainty about outcomes.
- Advanced Models: Explores incorporating coreference and structured event representations for more coherent generation.
- State-Space Models: Investigates state-space approaches for modeling beliefs and anticipating longer-term future states, aligning better with some theories of suspense.
- Related Tasks: Suggests exploring related tasks like inverting suspense understanding for generating more suspenseful stories.
Summary updated