Quantifying the narrative flow of imagined versus autobiographical stories

Maarten Sap, Anna Jafarpour, Yejin Choi, Noah A. Smith, James W. Pennebaker, Eric Horvitz

TL;DR

Researchers have developed a new metric called sequentiality that uses Large Language Models (LLMs) to quantify the "flow" of a narrative. By analyzing thousands of stories, they discovered that imagined stories are significantly more predictable and follow standard patterns (schemas) more closely than real memories—though, over time, our real memories actually become more "imagined" as they undergo mental consolidation.

What Problem Does This Paper Address?

While humans are natural storytellers, computationally measuring the "flow" of a narrative has historically been difficult. Most existing tools rely on surface-level features like word frequency, sentiment analysis, or sentence embeddings. However, these don't capture the underlying cognitive difference between episodic memory (recalling specific, idiosyncratic details of an event) and schematic knowledge (relying on general expectations of how an event should unfold, like a wedding or a doctor's visit).

The authors seek to answer: Can we mathematically distinguish between a story someone actually lived versus one they just made up? And how does the structure of our own memories change as we retell them over months?

Methods

The core innovation is a metric called sequentiality. It leverages the probabilistic nature of LLMs (specifically GPT-3) to measure how much the preceding context of a story helps predict the next sentence.

The researchers compare two generative models:

  1. Topic-driven model: Calculates the likelihood of a sentence based only on the overall story topic.
  2. Contextual model: Calculates the likelihood of a sentence based on the topic plus all preceding sentences.

Sequentiality is the difference between these two. A high sequentiality score means the sentence was highly predictable given the previous context—it follows a "standard" flow. A low score suggests the sentence is a "salient event" or a specific detail that deviates from the expected narrative arc.

Datasets

The study utilizes the Hippocorpus dataset, which contains ~7,000 diary-like short stories. The stories are categorized into three types:

  • Recalled: Stories about a real personal experience written shortly after it happened.
  • Retold: The same authors retelling those real experiences 3–6 months later.
  • Imagined: Stories written by different people who were given a brief summary of a real event and asked to "fill in the blanks" as if it happened to them.

A subset of 240 stories also included human annotations labeling sentences as "major" or "minor" events and "surprising" or "expected."

Key Insights and Ideas

  • Imagination is Schematic: Imagined stories have the highest sequentiality. When we make things up, we tend to follow "scripts" or "schemas" (e.g., if you're writing about a restaurant, you're likely to mention ordering, then eating, then paying).
  • Memory is Idiosyncratic: Real memories have lower sequentiality. Life is messy and filled with specific, non-prototypical details that a language model (trained on general text) finds harder to predict.
  • The "Gist-ification" of Memory: Perhaps the most fascinating finding is that retold stories have higher sequentiality than fresh recalls. As time passes, our brains perform "memory consolidation," stripping away specific episodic details and replacing them with more abstract, schematic versions. We essentially "narrativize" our own lives, making our pasts more predictable the more we think about them.
  • Major Events Break the Flow: Sentences identified by humans as "major events" (plot twists) consistently showed lower sequentiality. These are the "surprises" that deviate from the expected schema.

Key Results

  • Sequentiality Ranking: Imagined > Retold > Recalled. This confirms that imagined stories flow in a more "expected" manner than autobiographical ones.
  • Effect Size: Sequentiality was a much stronger predictor of story type than traditional metrics like word counts or "realis" (concrete event) detection.
  • Memory Decay: Retold stories were shorter and contained fewer "realis" events (concrete references to things that happened) compared to fresh recalls, supporting the idea that memories become more "gist-like" over time.
  • Surprise Factor: Sentences annotated as "surprising" had significantly lower sequentiality than "expected" ones, validating that the LLM-based metric aligns with human perception of narrative shifts.

Limitations and Future Work

The authors acknowledge that sequentiality is dependent on the LLM used. Because GPT-3 was trained primarily on Western, English-language internet text, the "expected flow" it identifies is culturally specific. A story that seems "low sequentiality" (unpredictable) to GPT-3 might be perfectly predictable to someone from a different culture.

Future research could apply this metric to:

  • Lie Detection: Identifying if a statement is "too schematic" to be a real memory.
  • Misinformation: Understanding how stories change as they are retold across social networks.
  • Cognitive Health: Using narrative flow as a potential marker for memory-related disorders like Alzheimer's, where the ability to recall episodic detail diminishes.

Summary updated

Abstract

Lifelong experiences and learned knowledge lead to shared expectations about how common situations tend to unfold. Such knowledge of narrative event flow enables people to weave together a story. However, comparable computational tools to evaluate the flow of events in narratives are limited. We quantify the differences between autobiographical and imagined stories by introducing sequentiality, a measure of narrative flow of events, drawing probabilistic inferences from a cutting-edge large language model (GPT-3). Sequentiality captures the flow of a narrative by comparing the probability of a sentence with and without its preceding story context. We applied our measure to study thousands of diary-like stories, collected from crowdworkers, about either a recent remembered experience or an imagined story on the same topic. The results show that imagined stories have higher sequentiality than autobiographical stories and that the sequentiality of autobiographical stories increases when the memories are retold several months later. In pursuit of deeper understandings of how sequentiality measures the flow of narratives, we explore proportions of major and minor events in story sentences, as annotated by crowdworkers. We find that lower sequentiality is associated with higher proportions of major events. The methods and results highlight opportunities to use cutting-edge computational analyses, such as sequentiality, on large corpora of matched imagined and autobiographical stories to investigate the influences of memory and reasoning on language generation processes.
BibTeX
@inproceedings{sap2022quantifying,
  author = {Maarten Sap and Anna Jafarpour and Yejin Choi and Noah A. Smith and James W. Pennebaker and Eric Horvitz},
  title = {{Quantifying the narrative flow of imagined versus autobiographical stories}},
  booktitle = {Proceedings of the National Academy of Sciences},
  year = {2022},
  doi = {10.1073/pnas.2211715119},
}

Loading related content...

Figure