Abstract
Lifelong experiences and learned knowledge lead to shared expectations about how common situations tend to unfold. Such knowledge of narrative event flow enables people to weave together a story. However, comparable computational tools to evaluate the flow of events in narratives are limited. We quantify the differences between autobiographical and imagined stories by introducing sequentiality, a measure of narrative flow of events, drawing probabilistic inferences from a cutting-edge large language model (GPT-3). Sequentiality captures the flow of a narrative by comparing the probability of a sentence with and without its preceding story context. We applied our measure to study thousands of diary-like stories, collected from crowdworkers about either a recent remembered experience or an imagined story on the same topic. The results show that imagined stories have higher sequentiality than autobiographical stories and that the sequentiality of autobiographical stories increases when the memories are retold several months later. In pursuit of deeper understandings of how sequentiality measures the flow of narratives, we explore proportions of major and minor events in story sentences, as annotated by crowdworkers. We find that lower sequentiality is associated with higher proportions of major events. The methods and results highlight opportunities to use cutting-edge computational analyses, such as sequentiality, on large corpora of matched imagined and autobiographical stories to investigate the influences of memory and reasoning on language generation processes.
Introduction
When humans construct narratives, they draw upon two primary cognitive resources: schemas (general knowledge of typical event sequences) and episodic memory (specific, unique details of actual experiences). While the qualitative differences between real memories and imagined stories are well-noted in cognitive science, quantifying the "flow" of these narratives has historically been challenging. This research introduces a computational approach using Large Language Models (LLMs) to measure these structural differences, treating the model as a normative baseline for common-sense expectations.
Methods
The researchers developed a metric called sequentiality to quantify narrative flow. Using GPT-3 (175B parameters), the metric calculates the difference between the probability of a sentence given only the story's topic versus its probability given both the topic and the preceding context. High sequentiality indicates a story follows a predictable, schematic "script," while low sequentiality suggests deviations driven by unique episodic details.
The study utilized the Hippocorpus dataset, containing approximately 7,000 diary-like stories across three categories:
- Recalled: Real personal experiences from the last 3–6 months.
- Retold: The same experiences retold after an additional 3–6 months.
- Imagined: Fictional narratives written based on summaries of the real stories.
Key Results
The analysis revealed several significant patterns in how humans structure memory versus imagination:
- Imagination is Schematic: Imagined stories consistently showed higher sequentiality than autobiographical ones ($p < 0.001$). When making stories up, humans rely more heavily on prototypical schemas, making their narratives more predictable to the LLM.
- Memory Consolidation: As memories age (moving from "Recalled" to "Retold"), their sequentiality increases. This suggests that over time, specific episodic details fade and are replaced by general schematic knowledge, making old memories structurally similar to imagined stories.
- Event Saliency: "Major events" (significant plot shifts) were found to lower sequentiality significantly, whereas minor events caused only local, temporary dips in predictability.
Discussion and Future Work
This study demonstrates that LLMs can serve as effective "computational lenses" for cognitive science. By identifying where human narratives diverge from a model's probabilistic distribution, researchers can pinpoint the influence of episodic memory.
Future applications of this work include forensic linguistics—such as identifying false confessions or deceptive reporting—and studying the structural differences between factual news and misinformation. However, the authors note that cultural biases in LLM training data may influence what the model considers an "expected" narrative flow, a factor that warrants further investigation in diverse linguistic contexts.
Summary updated