Suspense in Short Stories is Predicted By Uncertainty Reduction over Neural Story Representation

David Wilmot, Frank Keller

Introduction: The Mechanics of a "Page-Turner"

While Natural Language Processing (NLP) has made massive strides in generating coherent text, making that text compelling remains a "black box." In narratology, suspense is the engine of engagement, yet we lack a robust computational way to measure it. This research addresses a fundamental debate in psychology and linguistics: Is suspense caused by surprise (a backward-looking shock at what just happened) or uncertainty (a forward-looking anxiety about what might happen)?

Most previous NLP work focused on the former. This paper seeks to formalize these psychological concepts into a neural framework to see which one actually aligns with how humans experience dramatic tension. The authors propose that suspense is best modeled as "uncertainty reduction"—a measure of how much a new sentence narrows down the possible directions a story could take.

Methods: A Hierarchical Approach to Narrative

The authors propose a hierarchical language model designed to "read" a story sentence by sentence, maintaining a rolling representation of the narrative state. The architecture operates across three distinct levels:

  1. Word Level: Uses a pre-trained GPT (Generative Pre-Training) model to encode individual words into high-dimensional vectors.
  2. Sentence Level: An RNN aggregates these word embeddings into a single vector ($\gamma_i$) representing the semantic content of each sentence.
  3. Story Level: A second RNN processes these sentence vectors to create a contextualized story embedding ($e_t$), capturing the cumulative state of the narrative.
The hierarchical model architecture, showing the flow from word-level GPT embeddings to story-level RNN representations.

Measuring Suspense via "Rollouts"

To calculate the "forward-looking" measure of uncertainty reduction, the model performs a rollout. It uses beam search to generate multiple possible future continuations of the story from the current state. If a new sentence significantly reduces the variety (entropy) of those future paths, the model flags it as highly suspenseful.

The study compared two main frameworks for this calculation:

  • The Hale Model (Probability-based): Suspense is measured via the probability of the current state or the entropy of future states.
  • The Ely Model (Representation-based): Suspense is measured as the distance between vectors in the neural embedding space.

Key Insights: Forward-Looking Vectors

The research yielded several critical insights into how we perceive narrative tension:

  • Forward-Looking > Backward-Looking: Uncertainty reduction (looking ahead) is a much better predictor of suspense than surprise (looking back). Suspense isn't just about being shocked; it's about the narrowing of possibilities toward a high-stakes outcome.
  • Vectors Beat Probabilities: Measuring change in the neural representation space (the Ely model) was more effective than measuring changes in raw probability distributions. This suggests that the "meaning" captured in embeddings is more relevant to human suspense than the statistical likelihood of specific words appearing.
  • Sentiment Matters: Weighting the model based on emotional valence improved performance. Negative consequences are often perceived as more suspenseful than positive ones, aligning with "prospect theory" in economics.

Key Results: Near-Human Performance

The model was tested on the WritingPrompts dataset, a collection of ~300,000 short stories from Reddit, with human sentence-by-sentence suspense judgments collected via Amazon Mechanical Turk.

On the test set, the best model ($U^{Ely}$, uncertainty reduction over representations) achieved a Spearman’s correlation ($\rho$) of 0.711. For context, the human upper bound (inter-annotator agreement) was between 0.652 and 0.711, meaning the model is essentially performing at human levels of accuracy in identifying dramatic peaks.

Main results showing Spearman correlation (ρ) and Kendall’s tau (τ) for various models against human judgments.

Furthermore, the model demonstrated Zero-Shot Turning Point Detection. Without being specifically trained on movie data, its suspense peaks successfully predicted turning points in the TRIPOD dataset (movie synopses), outperforming several specialized baselines.

Limitations and Future Work

Despite its success, the model faces challenges. Calculating uncertainty reduction requires "rollouts"—generating many possible futures for every sentence—which is computationally expensive. Additionally, the model lacks a deep understanding of high-level plot structures (like long-term character goals), likely picking up on local textual cues that correlate with drama.

Future directions include moving toward variational methods (like state-space models) to model beliefs about future states more efficiently and investigating the model's vulnerability to adversarial inputs, such as shuffled sentences that retain "suspenseful" vocabulary without a coherent plot.

Summary updated

BibTeX
@inproceedings{wilmot2020suspense,
  author = {David Wilmot and Frank Keller},
  title = {{Suspense in Short Stories is Predicted By Uncertainty Reduction over Neural Story Representation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.161},
}
Figure